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Navy/Thomas  Nelson  Community 
College  MLT  Training 
Pilot  Evaluation 


cna| 


In  January  1999,  the  Navy  contracted  out  a  portion  of  its  enlisted 
Medical  Laboratory  Technician  (MLT)  training  to  Thomas  Nelson  Community 
College  (TNCC)  in  Hampton,  VA.  This  marked  the  beginning  of  a  2-year 
MLT  training  pilot  study  (ending  August  2001). 

During  the  development  of  the  pilot,  MED-05,  Director  of  Education, 
Training,  and  Personnel  for  the  Bureau  of  Medicine  and  Surgery,  Department 
of  the  Navy  (BUMED),  asked  CNA  to  develop  an  evaluation  plan  for  the  pilot. 
To  accomplish  this,  CNA  worked  closely  with  staff  members  from  the  Naval 
School  of  Health  Sciences  (NSHS)  Portsmouth  (Navy  oversight  for  the  TNCC 
MLT  program),  NSHS  San  Diego  (in-house  MLT  training  control  site  for  the 
pilot),  and  MED-05.  The  work  involved  identifying  key  parameters  for 
measuring  the  success  of  the  pilot  program  and  developing  appropriate  survey 
tools  and  additional  data  needed  to  (1)  monitor  the  ongoing  progress  of  the 
TNCC  MLT  program  and  (2)  evaluate  the  ultimate  success  of  the  pilot 
program  relative  to  in-house  training,  in  terms  of  actual  cost-savings  and 
training  output  (e.g.,  student  performance).  The  resulting  evaluation  plan  was 
documented  in  a  July  1999  CNA  Annotated  Briefing  (CAB).1 

Throughout  the  pilot,  CNA  continued  to  work  in  conjunction  with 
NSHS  San  Diego,  NSHS  Portsmouth,  and  MED-05  to  collect  data  in  support 
of  the  evaluation  plan.  This  briefing  documents  the  results  of  our  evaluation 
analysis. 


1 .  Cori  R.  Rattelman,  Navy/Thomas  Nelson  Community  College  MLT  Training  Pilot: 
Evaluation  Plan,  CAB  99-59,  July  1999. 
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Summary  of  Findings 


•  We  evaluated  the  TNCC  pilot  program  against  in- 
house  training  programs  for  six  broad  “measures 
of  success” 

•  Our  results  indicate  that  the  TNCC  program  was 
associated  with 

-  Producing  higher  quality  graduates  at  a  lower  cost  per 
graduate 

-  Estimated  savings  of  6%  to  15%  per  graduate 

•  This  was  achieved  while  maintaining 

-  Student  satisfaction  with  the  MLT  program  and  many 
aspects  of  military  quality  of  life 

-  Military  bearing 

-  End-user  satisfaction 


We  evaluated  the  TNCC  MLT  pilot  across  six  broad  areas:  quality  of 
graduates,  production  of  graduates,  cost,  student  satisfaction,  maintaining 
military  bearing  among  students,  and  end-user  satisfaction.  For  each  area, 
outcomes  from  the  TNCC  pilot  were  compared  with  outcomes  of  a  selected 
control,  or  comparison,  group  (particular  Navy  in-house  training  programs).1 

Our  findings  show  that  the  TNCC  program  produced  high-quality 
graduates  at  a  lower  per-graduate  cost  when  compared  with  the  in-house  MLT 
training  program.  Excluding  student  salaries  and  controlling  for  differences  in 
student  population  between  the  TNCC  and  in-house  program,2  we  found  that 

(continued) 


1 .  For  this  analysis,  control  group  refers  to  a  specific  Navy  in-house  training  program  (NSHS  SD 
MLT)  or  set  of  in-house  programs  (NSHS  Portsmouth  X-ray  Tech  and  Pharmacy  Tech).  These 
programs  are  meant  to  represent  a  comparison  point  when  evaluating  specific  outcomes.  The 
control  sites  are  not  expected  to  be  identical  to  the  experimental  site  (TNCC)  in  terms  of 
population  characteristic  and/or  program  design  (in  fact,  the  pilot  was  designed  to  allow  the 
civilian  contractor  to  propose  a  program  that  may  differ  significantly  in  design  from  the  in-house 
MLT  program,  see  pp.  5  -8  for  details).  We  use  regression  analysis  to  control  for  differences  in 
student  population  characteristics  between  the  experimental  and  control  groups.  Differences  in 
program  design  reflect  the  “treatment”  received  by  the  experimental  group. 

2.  Enrollee  placement  in  the  MLT  programs  was  not  random.  Compared  to  TNCC  MLT 
enrollees,  enrollees  in  the  in-house  MLT  program  were  more  likely  to  be  male,  to  be  junior 
pay  grade,  and  to  have  come  directly  from  Hospital  Corps  School,  and  less  likely  to  have  met 
prerequisites  and  to  have  lower  Corps  School  GPAs.  Again,  using  regression  analysis,  we 
attempt  to  statistically  control  for  differences  in  population  demographics  that  might 
independently  affect  the  outcome.  This  allows  us  to  isolate  the  impact  of  the  “treatment”  on 
student  outcomes.  A  discussion  of  this  methodology  follows  in  the  background  section  of  this 
briefing. 


Summary  of  Findings 


•  We  evaluated  the  TNCC  pilot  program  against  in- 
house  training  programs  for  six  broad  “measures 
of  success” 

•  Our  results  indicate  that  the  TNCC  program  was 
associated  with 

-  Producing  higher  quality  graduates  at  a  lower  cost  per 
graduate 

-  Estimated  savings  of  6%  to  15%  per  graduate 

•  This  was  achieved  while  maintaining 

-  Student  satisfaction  with  the  MLT  program  and  many 
aspects  of  military  quality  of  life 

-  Military  bearing 

-  End-user  satisfaction 


(continued) 

TNCC  cost  per  graduate  was  at  least  6  percent  lower  for  TNCC — if  one  included 
the  cost  of  overhead  provided  by  NSHS  Portsmouth  (costs  that  the  contractor 
cannot  control).  If  one  looked  only  at  MLT  course- specific  costs — those  costs 
over  which  the  contractor  does  have  control — we  estimate  that  TNCC  cost  per 
graduate  is  as  much  as  15  percent  lower  than  the  in-house  MLT  program.3 

In  addition,  the  TNCC  program  accomplished  this  while  continuing  to  be 
at  least  as  successful  as  the  in-house  programs  with  regard  to  attrition  rates, 
student  satisfaction  with  the  program  and  most  aspects  of  QOL,  incidence  of 
disciplinary  action  among  students,  maintaining  military  bearing,  and  the 
satisfaction  of  MTF  lab  supervisors  working  with  the  program’s  graduates. 

In  this  briefing,  we  will  walk  you  through  the  evaluation  plan, 
methodology,  data,  and  results  for  each  of  these  areas.  Let’s  begin  with  some 
background  on  the  Navy’s  MLT  training  program  and  the  development  of  the 
pilot. 


3.  We  estimate  the  total  cost  of  MLT  training  (at  TNCC  and  NSHS  San  Diego)  over  the  course  of 
the  pilot  evaluation  to  be  about  $5.9  million — $3,046,000  for  NSHS  SD  and  $2,856,000  for 
TNCC.  This  cost  excludes  student  salaries  (the  largest  portion  of  MLT  training  costs).  Appendix 
E  provides  a  complete  cost  breakout. 
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Background 


Background: 

Then  and  Now 

•  Historically,  MLT  was  taught  at  NSHS 
Bethesda  and  NSHS  San  Diego 

•  For  the  duration  of  the  2 -year  MLT  pilot: 

-  NSHS  Bethesda’ s  MLT  program  has  been 
temporarily  closed,  pending  the  pilot’ s  outcome 

-  MLT  students  were  sent  to  NSHS  San  Diego  and 
Thomas  Nelson  Community  College  (TNCC) 

-  NSHS  Portsmouth  provided  oversight  for 
TNCC’s  MLT  program 


Before  the  pilot  program,  Navy  Medicine  trained  about  130  MLTs 
annually.  This  training  took  place  at  two  sites:  NSHS  Bethesda  and  NSHS  San 
Diego.1  Each  of  these  schoolhouses  is  collocated  with  a  Navy  Medical  Center, 
NMC  Bethesda  and  NMC  San  Diego,  respectively. 

The  Navy’s  in-house  MLT  course  runs  369  days  with  instruction  5 
days  a  week  (courses  start  about  every  3  to  4  months).  The  course  is  divided 
into  quarters.  The  first  quarter  consists  of  didactic  instruction  and  is  followed 
by  clinical  lab,  didactic,  and  lab.  Didactic  instruction  is  provided  in  the 
schoolhouses  by  Navy  enlisted  MLTs  (supervised  by  the  department  head — a 
Navy  Lab  Officer).  The  course  quarters  dedicated  to  clinical  lab  are  conducted 
at  the  respective  Navy  Medical  Centers  under  the  supervision  of  the  hospital’s 
laboratory  supervisor.2 

MED-05  decided  to  temporarily  close  NSHS  Bethesda’ s  MLT  program 
(its  east  coast  training  site)  pending  the  outcome  of  the  pilot.  Lor  the  duration 
of  the  pilot,  it  was  determined  that  Bethesda’ s  projected  student  load  (50  to  60 
students  annually)  would  attend  the  TNCC  program  with  Navy  oversight 
provided  by  NSHS  Portsmouth  and  that  NSHS  San  Diego  would  continue  to 
train  MLT  students  on  the  west  coast. 


1 .  NSHS  San  Diego  had  a  higher  annual  throughput — approximately  7 1  students  per  year  vs. 
61  students  at  Bethesda  (based  on  3  -year  average  input,  1996-1998). 

2.  Clinical  lab  consists  of  rotation  through  each  of  the  lab’s  clinical  benches. 
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Background 


Background: 

Developing  the  Pilot  Program 

•  Allow  civilian  institution  to  develop  and 
own  a  unique  academic  curriculum  for  the 
Navy 

-  Not  required  to  duplicate  the  in-house  program 

-  Must  meet  certain  requirements 

•  Navy  still  wants  to: 

-  Produce  a  sailor  as  well  as  an  MLT 

-  Ensure  adequate  quality  of  life  for  Navy 
students  attending  a  civilian  training  program 


When  BUMED  decided  to  contract  out  a  portion  of  its  MLT  training,  it 
chose  to  give  the  bidding  civilian  institutions  as  much  flexibility  as  possible  in 
designing  their  proposed  academic  programs.  BUMED  established  the  following 
requirements  for  the  civilian  institutions: 

•  Produce  Navy  MLTs  within  a  fixed  time  frame  (369  days). 

•  Provide  capacity  for  a  maximum  of  96  students  per  year. 

•  Maintain  MLT  program  accreditation. 

•  Allow  curriculum  time  for  Navy-specific  training  taught  by  AD  instructors. 

•  Conduct  clinical  lab  portions  of  the  curriculum  only  at  NMC  Portsmouth. 

•  Provide  office  space  for  Navy  instructors  and  department  head. 

Each  bidder  could  decide  how  to  meet  these  requirements.  This  meant 
that  the  winning  civilian  institution  would  own  the  academic  curriculum  and  that 
the  resulting  program  might  look  significantly  different  from  the  in-house 
program  (in  terms  of  student/teacher  ratios,  matriculation  plans,  didactic/clinical 
rotations,  etc.).  Continued  program  accreditation,  as  well  as  an  incentive 
structure  based  on  graduation  rates,  pass  rates  on  the  national  MLT  certification 
exam  (the  ASCP  exam),  and  student  satisfaction,  would  ensure  that  the  resulting 
program  met  the  Navy’s  requirements  for  producing  Navy  MLTs.1 

(continued) 


1 .  To  provide  the  correct  incentive  structure,  the  incentive  fees  for  graduation  and  ASCP  exam 
pass  rates  were  tied  together  (small  incentive  for  graduation  rate  and  much  larger  incentive  for 
percentage  of  graduates  that  passed  the  ASCP). 
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Background 


Background: 

Developing  the  Pilot  Program 

•  Allow  civilian  institution  to  develop  and 
own  a  unique  academic  curriculum  for  the 
Navy 

-  Not  required  to  duplicate  the  in-house  program 

-  Must  meet  certain  requirements 

•  Navy  still  wants  to: 

-  Produce  a  sailor  as  well  as  an  MLT 

-  Ensure  adequate  quality  of  life  for  Navy 
students  attending  a  civilian  training  program 


(continued) 

In  addition,  BUMED  was  concerned  that  contracting  out  training 
should  not  limit  its  ability  to  (1)  produce  sailors  as  well  as  MLTs  or  (2) 
provide  adequate  quality  of  life  for  the  Navy  student  attending  a  civilian 
training  program  (rather  than  being  trained  at  a  Navy  schoolhouse). 

To  minimize  any  negative  impact  that  contracting  might  have  on  these 
goals,  the  pilot  implementation  team1  established  several  key  parameters  for 
the  pilot  program,  including  the  following: 

•  NSHS  military  staff  are  on  site  at  civilian  institution. 

•  NSHS  instructors  provide  Navy-specific  training  (e.g.,  Navy  Blood 

Program)  at  the  civilian  institution. 

•  Classes  move  through  the  program  as  a  military  unit. 

•  Students  are  in  uniform. 

•  Attending  group  PT  sessions  conducted  by  military  staff  is  mandatory. 

•  Use  NMC  Portsmouth  barracks  and  facilities. 

•  NSHS  Portsmouth  provides  transportation  to  contractor’s  facilities. 

1 .  The  implementation  team  consisted  of  members  from  MED-05  (HM/DT  Training  Programs, 
Accreditation,  Education  and  Training  Analysis,  and  Evaluation  Manager),  MLT  department 
heads  from  NSHS  San  Diego,  Bethesda,  and  Portsmouth  (newly  appointed),  NSHS 
Portsmouth  Command  staff,  NMC  Portsmouth  Laboratory  Manager,  HM/DT  Community 
Manager  and  staff,  detailing  community,  NAVMEDLOGCOM  Contracting,  and  CNA. 
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Background 


Background: 

TNCC’s  MLT  Program 

•  How  does  the  resulting  TNCC  MLT 
program  differ  from  the  in-house  program? 

-  Curriculum  (timing  and  content) 

-  Philosophy  (education  vs.  training) 

-  Education  and  experience  of  instructors 

•  Professors  from  a  variety  of  disciplines 

-  Cannot  use  “set  backs”  for  remediation 

-  Can/does  make  changes  continuously  to  adapt 
to  needs  of  students 


The  contract  for  the  MLT  pilot  program  was  awarded  to  Thomas 
Nelson  Community  College  in  July  1998,  with  a  start  date  of  January  1999. 
Based  on  the  academic  program  proposed  by  TNCC,  the  college  would  start  a 
class  every  6  months  for  the  duration  of  the  contract  (a  total  of  four  classes). 
The  TNCC  MLT  course  would  differ  from  the  in-house  program  in  several 
ways. 

As  noted  previously,  the  pilot  would  be  limited  to  a  course  length  of 
369  days  and  have  the  capacity  to  enroll  up  to  96  students  per  contract  year. 
TNCC’s  academic  program  is  taught  as  6  months  of  didactic  instruction  at  the 
college  campus,  followed  by  6  months  of  clinical  lab  at  NMC  Portsmouth 
(under  the  supervision  of  the  hospital’s  laboratory  manager) — as  opposed  to 
alternating  quarters  of  didactic  and  lab  modules. 

Didactic  instruction  is  provided  primarily  by  the  college  faculty 
(civilian  professors  from  a  variety  of  disciplines — mathematics,  chemistry, 
biology,  etc.),  with  the  exception  of  Navy-specific  coursework,  such  as  the 
Navy  Blood  Program  and  Military  Chemistry.  Navy-specific  instruction 
consists  of  about  32  hours  and  is  taught  by  the  onsite  Navy  staff. 

The  TNCC  curriculum  contains  much  of  the  same  content  as  the  in- 
house  MLT  program  (much  of  this  is  mandated  by  the  national  accreditation 
process  undertaken  by  both  programs).  The  TNCC  program,  however,  does 
include  additional  material,  such  as  a  technical  writing  course,  prep  and  review 
for  the  standardized  ASCP  national  MLT  certification  exam,  Internet  and 
(continued) 
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Background 


Background: 

TNCC’s  MLT  Program 

•  How  does  the  resulting  TNCC  MLT 
program  differ  from  the  in-house  program? 

-  Curriculum  (timing  and  content) 

-  Philosophy  (education  vs.  training) 

-  Education  and  experience  of  instructors 

•  Professors  from  a  variety  of  disciplines 

-  Cannot  use  “set  backs”  for  remediation 

-  Can/does  make  changes  continuously  to  adapt 
to  needs  of  students 


(continued) 

library  research  projects,  and  an  emphasis  on  “critical  thinking  skills.”  This 
parallels  the  college’s  overall  education  philosophy. 

TNCC  has  a  proposed  remediation  plan  for  students  who  fall  behind. 
The  plan  and  any  process  is  reviewed  periodically  by  the  onsite  Navy 
Department  Head/Contracting  Officer.  Remediation  processes  at  NSHS  San 
Diego  include  the  use  of  “set  backs.”  A  student  who  falls  behind  for  either 
academic  or  nonacademic  reasons  can  be  set  back  provided  it  is  determined 
that  he/she  (1)  cannot  successfully  complete  the  current  MLT  course  but  (2) 
has  the  potential,  with  either  appropriate  remediation  and/or  just  a  new  start,  to 
successfully  complete  the  next  MLT  course.  In  this  case,  the  student  attrites 
from  his/her  current  MLT  course  and  waits  for  the  start  of  the  next  course 
(recall  that  MLT  courses  at  NSHS  SD  start  about  every  3  to  4  months).  For  the 
pilot,  the  set-back  option  was  not  available  to  TNCC.  Faced  with  a  faltering 
student,  TNCC  staff  had  only  the  option  of  providing  remediation  or  failing 
(disenrolling)  the  student. 

Because  TNCC  “owns”  the  academic  curriculum,  it  can  make  changes 
at  will  to  adapt  to  the  needs  of  the  students,  such  as  providing  remedial 
lessons,  reviewing  lessons/course  modules,  and  changing  and  reworking 
module  content  and  timing  in  response  to  student  surveys  and  performance. 
Alternatively,  the  NSHS  MLT  program  is  more  fixed.  Changing  or  altering  the 
curriculum  in  any  way  requires  approval  through  the  chain  of  command,  up  to 
MED-05. 

It  is  important  to  recognize  these  program  differences  as  we  go  through 
our  analysis. 
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Background 


Background: 

MLT  Pilot  Evaluation  Plan 


•  For  select  “measures  of  success,”  compare 
outcomes  of  TNCC  pilot  program  and  control 
group 

•  Control  for  student  quality  and  demographics 

•  Cannot  control  for  individual  differences  in 
programs  (3-3 -3 -3  vs.  6-6,  student/teacher 
ratios,  active  duty  vs.  civilian  instructors,  etc.) 


In  the  course  of  developing  the  pilot  implementation  plan,  the  MLT 
working  group  identified  several  key  outcomes,  or  “measures  of  success,”  to 
determine  the  success  of  the  MLT  pilot  program. 

Both  to  monitor  the  TNCC  program  and  to  evaluate  the  pilot,  we 
measured  and  compared  these  selected  outcomes  for  the  TNCC  MLT  pilot 
program  and  selected  control  groups.  Depending  on  the  measure  of  success, 
the  appropriate  control  group  and  populations  differ.  For  example,  when 
comparing  the  quality  of  MLT  graduates,  we  want  to  look  at  TNCC  graduates 
relative  to  graduates  from  the  Navy’s  in-house  MLT  training  program  at 
NSHS  San  Diego.  Alternatively,  if  we  are  looking  at  quality-of-life  (QOL) 
issues,  we  want  to  compare  the  experiences  of  the  Navy  corpsmen  attending 
TNCC  with  the  experiences  of  Navy  corpsmen  attending  other  C-school 
courses  at  NSHS  Portsmouth.  That  is  because  the  QOL  goal  for  the  pilot 
program  was  to  have  the  Navy’s  TNCC  students  be  at  least  as  well  off  as 
NSHS  Portsmouth’s  students. 

To  accurately  measure  the  effect  of  the  TNCC  pilot  program  on  these 
outcomes,  we  must  control  for  those  individual  characteristics  that  might 
independently  affect  the  outcomes  (such  as  age,  gender,  marital  status,  years  of 
service,  and  student  quality).  For  example,  if  Navy  TNCC  students  are  more 
likely  than  those  at  NSHS  Portsmouth  to  be  married  with  dependents,  we 
might  find  that  satisfaction  with  some  aspects  of  QOL  differs  between  the  two 

(continued) 
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Background: 

MLT  Pilot  Evaluation  Plan 


•  For  select  “measures  of  success,”  compare 
outcomes  of  TNCC  pilot  program  and  control 
group 

•  Control  for  student  quality  and  demographics 

•  Cannot  control  for  individual  differences  in 
programs  (3-3-3-3  vs.  6-6,  student  teacher 
ratios,  active  duty  vs.  civilian  instructors,  etc.) 


(continued) 

groups.  This  difference  might  be  driven  by  the  abundance  or  lack  of  services  for 
families  rather  than  any  actual  differences  in  access  to  the  available  services  for 
these  two  groups. 

When  we  compare  outcomes  of  the  TNCC  program  and  the  control 
group,  we  cannot  individually  control  for  program- specific  differences,  such  as 
student/teacher  ratios,  matriculation  plans,  and  3-month  didactic/clinical 
rotations  (3-3-3-3)  vs.  6-month  rotations  (6-6).  These  program- specific 
characteristics  are  perfectly  correlated  with  the  location  of  the  program  (TNCC 
or  NSHS  San  Diego)  and  therefore  become  part  of  the  overall  program 
differences. 

The  next  set  of  slides  will  provide  more  detailed  information  on  the 
methodology  and  data. 
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Methodology 


•  For  each  measure  of  success,  we  estimate  the 
probability  of  an  outcome  as  a  function  of: 

-  Quality  and  demographic  characteristics 

-  Treatment  group  (TNCC  vs.  NSHS  SD  or  Portsmouth) 

-  Startup  (first  year  of  pilot  vs.  second  year) 

•  Statistical  significance  is  determined  by  the 
significance  test  on  the  treatment  control 

•  We  then  predict  the  mean  probability  of  an  outcome 
at  the  treatment  site  and  control  site  using  the  pooled 
population  sample 


For  each  of  the  selected  measures  of  success,  we  want  to  measure  the 
relative  difference  between  success  for  the  population  at  the  TNCC  MLT 
course  (the  treatment  group)  and  the  population,  for  example,  at  NSHS  San 
Diego’s  MLT  course  (the  control  group). 

Although  one  might  assume  random  selection  criteria  for  placing 
students  in  either  the  TNCC  or  NSHS  San  Diego  MLT  courses,  there  is  always 
concern  that  the  population  of  students  attending  each  of  these  courses  may 
systematically  differ  in  ways  (demographic  or  student  quality  characteristics) 
that  might  affect  their  performance  and/or  satisfaction  with  the  course. 
Therefore,  we  want  to  control  for  the  effects  of  these  characteristics  on  any 
outcome  measures  that  we  use  for  our  evaluation.  To  do  this,  we  use  a 
regression  model  that  explicitly  controls  for  differences  in  demographics  and 
student  quality  between  the  treatment  and  control  populations.  This  allows  us 
to  isolate  the  effect  of  attending  the  TNCC  MLT  course. 

Using  a  multivariate  logistic  regression  model,  we  estimate  the 
probability  of  an  outcome  (such  as  attrition  from  course)  as  a  function  of  the 
quality  and  demographic  characteristics  of  the  student  and  his/her  peers, 
whether  one  was  in  the  treatment  or  control  group,  and  a  startup  variable  (to 
distinguish  any  effects  associated  with  the  initiation  of  a  new  program — year 
one  vs.  year  two). 

(continued) 
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Methodology 


•  For  each  measure  of  success,  we  estimate  the 
probability  of  an  outcome  as  a  function  of: 

-  Quality  and  demographic  characteristics 

-  Treatment  group  (TNCC  vs.  NSHS  SD  or  Portsmouth) 

-  Startup  (first  year  of  pilot  vs.  second  year) 

•  Statistical  significance  is  determined  by  the 
significance  test  on  the  treatment  control 

•  We  then  predict  the  mean  probability  of  an  outcome 
at  the  treatment  site  and  control  site  using  the  pooled 
population  sample 


(continued) 

Whether  a  statistically  significant  difference  in  outcomes  exists 
between  the  two  populations  is  determined  by  a  significance  test  on  the 
coefficient  estimate  for  the  treatment  group. 

Using  the  results  from  the  regression  analysis,  we  are  then  able  to 
predict  how  specific  individuals  (with  their  unique  characteristics)  would 
perform  in  either  setting  (the  treatment  or  control  environment).  This  allows  us 
to  estimate  the  mean  probability  of  an  outcome  at  TNCC  and  the  appropriate 
control  site  using  the  pooled  population  sample. 

In  essence,  this  gives  us  the  mean  predicted  outcome  associated  with 
sending  the  pooled  population  to  TNCC  vs.  sending  the  same  population  to  the 
control  site  (either  NSHS  San  Diego  or  Portsmouth).  See  appendix  A  for  a 
more  detailed  discussion  of  the  methodology. 
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Methodology: 

Specific  Regression  Analysis  Controls 

•  Individual  student  quality 

—  AFQT 

—  GPA  at  Hospital  Corps  School  (entry  skills  training) 

—  Force  fill 

—  Previous  basic  MLT  training 
—  College  experiance 

—  Status  of  prerequisites  (math,  science,  no  waiver) 

—  Paygrade 

•  Peer  characteristics,  for  example 

—  Average  student  quality  valuables  for  one’s  class  peers 

((£;  for  j  AFQT; )  -  AFQT; )  /  (rij  -  1)  where  i  is  enrollee  in  class  j 

•  Demographics 

—  Gender,  age,  and  race/ethnicity 
—  Marital  status  and  number  of  children 


What  specific  characteristics  do  we  control  for  in  our  regression  analysis? 
The  exact  controls  used  for  each  measure  may  vary,  but  in  general  we  are 
attempting  to  control  for  the  quality  of  each  student,  the  quality  of  each  student’s 
peers,  and  individual  demographics. 

For  our  analysis,  individual  student  quality  can  be  captured  with  several 
measures.  AFQT  represents  a  general  measure  of  student  quality,  whereas  one’s 
Hospital  Corps  School  GPA  is  more  representative  of  one’s  quality  in  the 
specific  arena  of  health  care.  Whether  a  person  was  a  “force  fill,”  sent  directly 
from  Corps  School  to  the  MLT  program  rather  than  doing  a  minimum  of  one  tour 
of  duty  as  a  general  HM,  would  indicate  his/her  experience  in  the  health  care 
field  and  level  of  maturity.  Another  indication  would  be  whether  a  person  was 
previously  trained  as  a  basic  MLT.  We  also  have  information  regarding  each 
MLT  student’s  background  with  regard  to  previous  college  attendance,  whether 
he/she  has  met  the  math  and  science  prerequisites  for  the  course,  and  the 
student’s  paygrade. 

One’s  educational  experience  can  be  significantly  influenced  by  the 
characteristics  of  one’s  peers.  For  this  reason,  we  examined  the  effect  of  several 
peer  characteristics  on  outcomes.  Specifically,  for  each  of  the  student  quality 
measures,  we  calculated  the  average  student  quality  for  one’s  classmates. 

Finally,  we  control  for  several  demographic  characteristics  that  might 
affect  performance  and/or  satisfaction.  These  include  gender,  race/ethnicity, 
marital  status,  whether  the  student  has  children,  whether  family  members  reside 
with  him/her,  and  whether  the  student  lives  in  base  housing. 
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Methodology: 

Data  and  Data  Sources 

•  Data  sources  for  quality  and  demographic  controls 

—  TNCC  and  NSHS  SD  enrollment  data 

—  TNCC,  NSHS  SD,  and  NSHS  PTS  end-of-course 
surveys 

—  TNCC  end-of-didactic-phase  survey 
—  Enlisted  Master  File 

•  Additional  data  sources  include 

—  Post-graduation  telephone  survey 

•  Graduates  and  graduates’  supervisors 

•  Conducted  at  4  and  12  months  after  graduation 
—  Cost  data  from  Navy  schools  and  MED-5 1 


Our  analysis  required  that  we  collect  data  from  several  different  sources. 
Data  sources  for  quality  and  demographic  controls  came  primarily  from  data 
collected  at  the  schoolhouses.  Both  TNCC  and  NSHS  San  Diego’s  MLT 
programs  provided  us  with  enrollment  data  for  each  student,  and  graduates  at 
TNCC,  NSHS  San  Diego,  and  selected  NSHS  Portsmouth  courses  provided 
some  demographic  information  via  end-of-course  surveys.1  In  addition,  we 
matched  each  MLT  student  to  the  Enlisted  Master  File  (EMF)  and  pulled 
comparable  data.2  The  EMF  data  were  used  when  student  quality  and 
demographic  variables  for  a  student  were  not  available  through  another  source 
(enrollment  or  survey  data). 

Additional  data  sources  included  post-graduation  telephone  surveys  of 
graduates  and  their  immediate  lab  supervisors.  These  surveys  were  conducted  4 
months  and  12  months  after  graduation  and  provided  data  regarding  the 
graduate’s  satisfaction  with  the  course  (from  the  perspective  of  having  now 
worked  as  an  entry-level  lab  technician  in  a  hospital  or  clinic  setting)  and  the 
supervisor’s  satisfaction  with  the  graduate’s  performance  and  military  behavior. 

Finally,  cost  data  were  collected  from  the  individual  schoolhouses  and 
the  Resource  Division/Manpower  Office  for  the  Assistant  Chief  for  Education, 
Training,  and  Personnel  (MED-5 1). 


1.  NSHS  Portsmouth’s  X-ray  and  Pharmacy  Technician  courses  were  chosen  to  be  control 
groups  for  measures  of  success  dealing  with  student  satisfaction  with  QOL.  These  courses  were 
chosen  based  on  course  length,  student  load,  and  anticipated  student  characteristics. 

2.  MLT  students  were  matched  to  EMF  data  for  the  quarter  in  which  they  started  the  course. 
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Measures  of  Success 

•  Quality  of  product 

•  Production  success 

•  Cost  efficiency 

•  Student  satisfaction 

•  End-user  satisfaction 

•  Military  bearing 


The  MLT  working  group  identified  and  defined  the  measures  of 
success  listed  above  and  outcomes  associated  with  each.  In  some  cases,  there 
may  be  several  outcomes  associated  with  a  single  measure  of  success  (for 
example,  students  were  surveyed  regarding  their  satisfaction  with  16 
different  aspects  of  the  MLT  course  and  15  aspects  of  military  quality  of 
life). 

Having  defined  six  broad  measures  of  success,  we  have  to  ask  the 
following  questions:  What  if  the  pilot  program  does  not  achieve  all  of  these 
goals?  How  will  Navy  Medicine  determine  the  ultimate  success  of  the 
program  if  some  of  the  measures  indicate  success  and  others  do  not?  Are 
some  more  important  than  others? 

Navy  Medicine  decided  that  cost  savings  subject  to  at  least 
maintaining  the  same  graduate  quality  as  NSHS  San  Diego  should  be 
considered  the  most  important  criterion  for  determining  the  success  of  the 
MLT  pilot.  The  other  criteria — student  satisfaction  with  the  program  and 
QOL,  militarization,  and  end-user  satisfaction — should  be  used  to  signal  the 
need  for  program  changes  and/or  enhancements  rather  than  to  determine 
ultimate  success  or  failure. 
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Primary  Measures 


•  Quality  of  product 

-  TNCC  has  higher  pass  rate  on  national  registry  exam 

•  66%  vs.  37%  at  control  site 

•  Production  success 

-  TNCC  attrition  rates  at  least  equivalent  to  control  group 
(excluding  set  backs) 

-  In  some  cases  TNCC  attrition  much  lower  (including 
set  backs,  10%  vs.  29%  at  control  site) 

•  Cost  efficiency 

-  TNCC  cost  per  graduate  is  lower  than  control  group 

•  6%  lower,  including  NSHS  overhead  in  model 

•  15%  lower,  excluding  NSHS  overhead  from  model 


To  determine  whether  the  pilot  achieved  cost  savings  subject  to  at 
least  maintaining  the  same  graduate  quality  as  NSHS  San  Diego  requires 
looking  at  three  separate  criteria:  quality  of  product,  production  success 
(which  is  used  to  determine  cost  per  graduate),  and  cost  efficiency.  We  refer 
to  these  as  the  “primary”  measures  of  success  for  this  evaluation. 

Ultimately,  we  found  that  quality  in  terms  of  pass  rate  on  a  national 
MLT  registry  exam  was  significantly  higher  for  TNCC  graduates  than  for 
graduates  from  the  in-house  MLT  program.  In  terms  of  production  success, 
we  found  that  the  academic  attrition  rate  was,  at  a  minimum,  no  different 
from  the  attrition  rate  at  NSHS  San  Diego;  in  some  cases  (if  one  included 
academic  set  backs  as  attrition),  it  was  significantly  lower. 

Using  predicted  attrition  rates  to  estimate  the  number  of  graduates  at 
TNCC  and  NSHS  San  Diego,  controlling  for  enrollee  characteristics,  we 
estimated  the  cost  per  graduate,  excluding  student  salaries.  We  found  that 
TNCC’s  per-graduate  cost  was  6%  to  15%  lower  than  that  of  NSHS  San 
Diego.  Therefore,  the  pilot  did  achieve  cost  savings  while  maintaining 
attrition  and  improving  the  quality  of  MLT  graduates. 
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Secondary  Measures 


•  Student  satisfaction 

-  Satisfaction  with  TNCC  program  equivalent  or  higher 
for  all  16  measures 

-  Satisfaction  with  QOL  at  TNCC  equivalent  or  higher 
for  11  of  15  measures 

•  End-user  satisfaction 

-  No  difference  between  programs  observed  4  months 
after  graduation 

•  Military  bearing 

-  No  difference  between  programs 


Although  militarization,  and  student  and  end-user  satisfaction,  will 
not  be  used  to  determine  the  ultimate  success  or  failure  of  the  MLT  pilot, 
they  do  reflect  significant  concerns  of  Navy  Medicine.  As  stated  previously, 
these  “secondary”  measures  of  success  have  been  (throughout  the  course  of 
the  pilot)  and  should  continue  to  be  used  to  focus  the  contractor  and  Navy 
Medicine  on  areas  for  program  improvements  and/or  enhancements. 

We  found  that,  for  the  most  part,  the  TNCC  program  was  successful 
at  maintaining  a  level  of  student  satisfaction  with  the  program  and  with  QOL 
equivalent  to  the  satisfaction  level  of  Navy  students  attending  in-house 
training  programs.  In  addition,  we  found  that  4  months  after  graduation, 
graduates’  immediate  lab  supervisors  were  just  as  satisfied  with  the 
knowledge,  technical  skills,  military  bearing,  attitude,  and  overall 
performance  of  TNCC  graduates  as  they  were  with  MLT  graduates  from 
NSHS  San  Diego.  We  will  continue  to  monitor  supervisor  and  graduate 
satisfaction  throughout  the  graduates’  first  12  months  as  MLTs. 

Starting  with  the  next  section,  we  will  address  each  of  these  six 
measures  of  success  in  turn,  discussing  in  detail  the  motivation,  outcomes, 
and  results  of  our  analysis. 
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Quality  of  Product 


•  How  do  we  measure  the  quality  of  TNCC 
MLT  graduates? 

•  Pass  rate  on  standardized  ASCP  exam 

•  Compare  with  NSHS  San  Diego  pass  rates 

-  Historical  pass  rates  are  not  relevant 

•  Although  ASCP  was  the  agreed -on  measure 
for  quality. . . 

-  Not  an  explicit  requirement/objective  for  the  in- 
house  program 


Certainly,  one  of  the  most  important  criteria  for  a  successful  training 
program  is  the  quality  of  the  graduates  it  produces.  Navy  Medicine  wants  to 
ensure  that  the  outsourcing  of  MLT  training  does  not  lead  to  a  decline  in  the 
quality  of  its  MLTs.  There  are  several  ways  to  measure  the  quality  of  the  MLT 
graduates,  but  the  working  group  concluded  that  the  most  objective  measure  of 
quality  would  be  performance  on  the  standardized  American  Society  of 
Clinical  Pathologists  (ASCP)  certification  exam.1 

For  the  duration  of  the  pilot,  Navy  MLT  graduates  from  both  TNCC 
and  NSHS  San  Diego  took  the  ASCP  exam  immediately  following  graduation. 
To  determine  whether  there  are  quality  effects,  we  compare  the  pass/fail  rates 
of  the  two  programs.2 

(continued) 


1 .  Although  ASCP  certification  for  civilian  technicians  is  not  required  by  individual  states,  the 
federal  government,  or  the  Clinical  Lab  Improvement  Act  (CLIA),  it  is  fair  to  say  that,  in 
general,  ASCP  certification  is  the  standard  of  practice  in  the  civilian  sector.  For  many  civilian 
hospitals,  particularly  in  states  with  no  state  licensure  requirement,  ASCP  certification  is  a 
condition  of  employment. 

Some  members  of  the  MLT  working  group  voiced  concerns  that,  if  a  single  test  were  used 
to  quantify  the  quality  of  graduates,  the  schools  might  simply  teach  to  that  exam.  The  group 
ultimately  decided  that  teaching  to  the  ASCP  exam  should  produce  a  competent  technician  and 
was  perfectly  acceptable. 

2.  The  decision  was  made  not  to  compare  current  pass  rates  with  those  from  past  Navy 
graduates.  In  the  past,  ASCP  was  not  mandatory.  Navy  graduates  could  choose  to  take  the  test 
on  their  own.  Those  who  chose  to  take  the  exam  were  more  likely  to  be  the  MLT  graduates 
who  believed  they  had  a  relatively  high  chance  of  passing  (a  self -selected  group). 
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Quality  of  Product 


•  How  do  we  measure  the  quality  of  TNCC 
MLT  graduates? 

•  Pass  rate  on  standardized  AS  CP  exam 

•  Compare  with  NSHS  San  Diego  pass  rates 

-  Historical  pass  rates  are  not  relevant 

•  Although  AS  CP  was  the  agreed -on  measure 
for  quality. . . 

-  Not  an  explicit  requirement/objective  for  the  in- 
house  program 


(continued) 

Even  though  members  of  the  Navy’s  Medical  Technologist  community 
agreed  that  the  ASCP  was  a  valid  measurement  of  quality  for  Navy  MLTs, 
passage  of  the  registry  exam  has  never  been  stated  as  the  requirement/ 
objective  for  the  in-house  program.  Therefore,  the  NSHS  SD  curriculum  has 
not  been  oriented  to  specifically  address  this  goal — note  that  TNCC  does 
include  a  prep  and  review  for  the  ASCP  exam.3  But,  it  is  our  understanding 
that  a  comprehensive  MLT  course  should  provide  the  basic  knowledge 
required  to  pass  the  ASCP  exam. 


3.  Prep  and  review  for  the  ASCP  included  retesting  TNCC  students  at  the  end  of  each  subject 
module  or  block  (all  course  tests  and  review  tests  are  modeled  after  the  ASCP)  and  a  final  exit 
exam  that  serves  as  a  mock  ASCP.  In  addition,  each  TNCC  student  received  the  ASCP  study 
guide  and  a  review  book  at  the  end  of  the  didactic  phase  (purchased  by  TNCC).  Finally,  at  the 
end  of  the  course,  students  participate  in  a  review  session  that  may  include  reviewing  their  old 
exams,  reviewing  material  as  requested,  and  having  instructors  available  for  question  and 
answer  sessions  (this  prep/review  session  has  had  no  specific  format;  rather,  it  was  adapted  to 
the  needs/desires  of  the  students  in  each  of  the  TNCC  pilot  courses).  All  prep  and  review  is 
conducted  within  the  allotted  training  time  frame,  and  all  costs  are  included  within  the  current 
contract  price. 
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Comparison  of  ASCP  Pass  Rates: 
NSHS  San  Diego  and  TNCC 

•  Population  is  MLT  program  graduates  who  have 
taken  the  ASCP  exam 

-  TNCC  (n=135) 

-  NSHS  SD  (n=121) 

•  Comparing  populations,  NSHS  SD  had: 

-  More  E-ls/E-2s  (23%  vs.  6%) 

-  More  force  fills  (60%  vs.  35%) 

-  More  basic  MLTs  (7%  vs.  3%) 

-  Fewer  women  (24%  vs.  36%) 

•  Predicted  ASCP  pass  rate  for  pooled  population 

-  TNCC  66%  NSHS  SD  37%  (statistically  sig.) 


The  population  of  interest  for  this  outcome  measure  is  MLT  graduates 
from  the  TNCC  and  NSHS  SD  program  who  have  sat  for  the  ASCP  exam.1  As 
part  of  the  pilot,  all  Navy  MLT  graduates  were  expected  to  sit  for  the  exam.2 

When  we  compare  the  characteristics  of  graduates  from  the  two 
programs,  we  find  that  NSHS  SD  had  more  graduates  in  junior  paygrades 
(started  the  program  as  an  E-l  or  E-2).  San  Diego  graduates  were  more  likely 
to  have  come  directly  from  Hospital  Corps  School  (force  fills).  We  also  found 
that  San  Diego  graduates  were  more  likely  to  have  previously  been  functioning 
in  the  fleet  as  a  basic  MLT  and  less  likely  to  be  female.  Otherwise,  there  were 
no  statistically  significant  differences  between  NSHS  SD  and  TNCC  graduates 
(such  as  AFQT,  Corps  School  GPA,  waivers  and  prerequisites,  previous 
college,  age,  marital  status,  or  children).3 

Controlling  for  quality  and  demographic  differences  of  graduates,  we 
find  that  graduation  from  TNCC  is  associated  with  a  higher  pass  rate  on  the 
ASCP  exam  (predicted  TNCC  pass  rate  is  over  75%  higher  than  NSHS  SD).4 


1 .  This  includes  graduates  from  MLT  courses  that  fall  within  the  time  frame  of  the  pilot 
evaluation  period.  See  appendix  B  for  a  list  of  eligible  courses.  This  list  applies  to  all  outcome 
measures  discussed  in  this  brief. 

2.  There  were  two  instances  in  which  graduates  were  not  able  to  participate — one  graduate 
each  from  the  TNCC  MLT  program  and  the  NSHS  SD  MLT  program. 

3.  See  appendix  C  for  ASCP  comparison  population  means  and  regression  results. 

4.  Raw  ASCP  pass  rates  (using  population  means,  not  adjusted  for  demographic  differences  in 
student  population  characteristics)  were  77%  for  TNCC  and  31%  for  NSHS  SD. 
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Production  Success 


•  Is  the  TNCC  program  successful  at 
producing  MLT  graduates? 

•  Examine  attrition  rates 

-  Disenrollments 

-  Set  backs 

•  Compare  to  NSHS  San  Diego  rates 


In  evaluating  the  success  of  a  training  program,  we  must  look  not  only 
at  the  quality  of  graduates  but  also  at  the  program’s  ability  to  produce 
graduates.  The  program’s  standards  may  be  very  high  and  its  graduates  of  the 
utmost  quality;  however,  if  the  program  attrites  most  of  its  students,  is  it  a 
success? 

We  examined  the  amount  and  types  of  attrition  from  the  TNCC 
program  and  compared  those  with  attrition  rates  at  NSHS  San  Diego.  Attrition 
rates  can  give  us  insight  into  the  training  program’s  process  for  dealing  with 
those  students  who  fall  behind  academically  because  of  problems  with  the 
course  material  and/or  because  of  personal  problems.  Navy  Medicine  believes 
that  its  in-house  programs  are  especially  good  at  dealing  with  such  problems . 
The  concern  was  that,  in  outsourcing  the  training  program,  Navy  Medicine 
would  lose  this  capability. 

Attrition  rates  can  also  have  implications  for  the  program’s  costs. 
Attrites  increase  the  cost  of  producing  MLT  graduates,  but  providing  all  the 
resources  necessary  to  eliminate  the  risk  of  academic  attrition  and/or  set  backs 
may  lead  to  exorbitant  program  cost.  We  will  examine  cost  issues  later  in  the 
briefing. 
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Defining  Attrition 

•  Disenrollments 

-  Academic 

-  Nonacademic 

•  Setbacks 

-  Academic 

-  Nonacademic 

•  Our  focus  is  academic  attrites  only 

•  Should  set  backs  be  included? 


Broadly  defined,  an  attrite  is  any  enrollee  who  did  not  successfully 
complete  the  MLT  course  (any  nongraduate).  Within  this  definition,  there  are: 

•  Academic  attrites ,  those  who  fail  to  successfully  complete  the  course 
because  they  are  unable  to  perform  at  the  required  academic  level 

•  Nonacademic  attrites ,  those  who  fail  to  successfully  complete  the  course 
for  reasons  other  than  academic  performance,  such  as  motivation, 
administrative,  legal,  medical,  or  personal  issues. 

For  this  analysis,  we  consider  the  occurrence  of  nonacademic  attrites  to 
be  random  (e.g.,  colorblindness,  death,  illness,  personal  and/or  administrative 
matters  that  interfere  with  the  person’s  ability  to  perform,  drug/alcohol- related 
offenses,  and  motivational  issues).  Therefore,  we  evaluate  TNCC’s  production 
success  (or  failure)  by  focusing  only  on  the  incidence  of  academic  attrition 
relative  to  that  of  NSHS  San  Diego  (again  controlling  for  differences  in 
student  quality  and  demographics). 

Another  issue  to  consider  is  how  to  deal  with  academic  set  backs.  Set 
backs  occur,  for  example,  when  a  person  is  struggling  in  the  MLT  class,  but  is 
judged  to  have  the  promise  of  successfully  completing  the  course  if  he/she  sits 
out  the  rest  of  the  quarter  (with  significant  remediation)  and  starts  the  next 
MLT  class  cycle  from  the  beginning.1  Set  backs  have  been  used  extensively  at 
SD,  but  TNCC  has  not  had  the  option,  making  it  hard  to  get  a  "pure 
comparable  measure  of  attrition."  As  a  result,  we  evaluate  academic  attrition 
both  including  and  excluding  set  backs. 


1 .  Other  examples  of  how  the  set-back  concept  might  be  applied  will  be  discussed  later. 
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Incidence  and  Types  of  Attrition  in 
Each  MLT  Program 

NSHS SD  TNCC 


Total  Enrollments*: 

202 

150 

Set  backs:  Academic 

42 

21% 

0 

- 

Medical 

0 

- 

0 

- 

Other  nonacademic 

1 

0.5% 

0 

- 

Disenrollments:  Academic 

22 

11% 

10 

7% 

Medical 

6 

3% 

3 

2% 

Other  nonacademic 

9 

4% 

1 

1% 

dque  Enrollments**: 

159 

150 

Disenrollments:  Academic 

22 

14% 

10 

7% 

Medical 

6 

4% 

3 

2% 

Other  nonacademic 

9 

6% 

1 

1% 

*  Set  backs  are  counted  as  two  separate  enrollments  (initial  enrollment  and  secondary  enrollment  in  subsequent  course). 
**  Set  backs  are  counted  only  once. 


We  begin  by  looking  at  the  incidence  and  types  of  attrition  that 
occurred  in  each  of  the  MLT  programs. 

In  the  table  above,  we  show  attrition  rates  (by  type  of  attrition)1  for  the 
population  of  total  enrollees  (this  includes  set  backs  as  attrites,  for  SD  n=202  ) 
and  for  the  population  of  unique  students  (this  excludes  set  backs  from  the 
population,  for  SD  n=159).2 

In  the  next  set  of  slides,  we  will  compare  attrition  rates  of  the  two 
programs,  controlling  for  individual  student  quality  and  demographic 
characteristics. 


1 .  “Other  nonacademic”  attrites  include  those  classified  as  (a)  personal  and/or  family 
emergencies  and  (b)  motivational,  attitude,  and  misconduct.  Of  SD's  nine  other  nonacademic 
disenrollments  (DEs),  one  was  personal  and  eight  were  motivational,  attitude,  or  misconduct. 
For  TNCC,  the  one  other  nonacademic  DE  was  classified  as  attitude/misconduct. 

2.  The  population  for  TNCC  (n=150)  is  the  same  whether  we  use  “total  enrollment”  or  “unique 
students”  because  TNCC  was  not  able  to  use  the  set-back  concept  and  therefore  had  no  set 
backs. 
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Comparison  of  Academic  Attrition  Rates: 
NSHS  San  Diego  and  TNCC 


•  Population  is  MLT  program  enrollees,  excluding 
nonacademic  attrites  (nonacademic  attrition  rate  6%) 

-  TNCC  (n=146) 

-  NSHS  SD  (n=186) 

•  Comparing  makeup  of  two  classes 

-  NSHS  SD  enrollees  had: 

•  Lower  Hospital  Corps  School  GPAs  (86.4  vs.  87.9) 

•  More  enrollees  missing  prerequisites  (10%  vs.  3%) 

•  More  force  fills  (64%  vs.  38%) 

•  More  E-ls/E-2s  (30%  vs.  10%) 

•  More  E-6s  (3%  vs.  1%) 

•  Fewer  females  (20%  vs.  36%) 


If  we  are  to  include  set  backs  as  academic  attrites,  we  start  with  the 
population  of  all  enrollments  to  either  the  NSHS  SD  or  TNCC  MLT  programs 
(anyone  who  starts  a  course),  excluding  nonacademic  attrites.1  Again,  we 
exclude  nonacademic  attrites  because  we  consider  these  cases  to  be  purely 
random.  It  is  merely  coincidence  that  a  colorblind  HM  was  enrolled  in  SD’s 
program  rather  than  TNCC  (colorblindness  is  an  automatic  disqualification  for 
an  MLT).  The  nonacademic  attrition  rate  for  the  pooled  population  of  enrollees 
was  6%  (16  at  NSHS  SD  and  4  at  TNCC).  This  leaves  us  with  an  enrollee 
population  of  332. 

Of  the  332  enrollees,  146  are  TNCC  program  enrollees  and  186  are 
NSHS  SD  enrollees.  The  programs  had  10  and  64  academic  attrites, 
respectively.  NSHS  SD’s  academic  attrites  were  split  between  set  backs  and 
disenrollments  (42  set  backs,  22  disenrollments). 

Comparing  this  population  of  enrollees  at  each  of  the  two  programs,  we 
find  that  NSHS  SD  had  fewer  female  enrollees.  Other  differences  were  in 
student  quality:  SD  enrollees  had  lower  Corps  School  GPAs  and  were  more 
likely  to  be  junior  paygrade  and  to  have  come  directly  from  Corps  School.2 


1 .  The  measure  “all  enrollments”  counts  set  backs  twice.  Even  though  the  set  back  represents 
one  unique  student  in  the  program,  that  student  is  enrolled  in  two  MLT  courses  before 
completion  of  the  program  (the  original  course  from  which  he/she  is  set  back  and  the  next 
available  course  from  which  he/she  will  ultimately  pass  or  fail). 

2.  See  appendix  D  for  academic  attrition  comparison  population  means  and  regression  results. 
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Comparison  of  Academic  Attrition  Rates: 

Results 


•  Attrition  ( including  set  backs  for  SD) 

-  Predicted  rate  for  pooled  population 

TNCC  10%  NSHSSD  29%  (statistically  sig.) 

•  Attrition  ( excluding  set  backs  for  SD  (n=40)) 

-  TNCC  (n=146) 

-  NSHS  SD  (n=144) 

-  Comparing  characteristics  across  two  classes,  GPA 
differential  disappears 

-  Controlling  for  individual  characteristics,  we  find  no 
statistically  significant  difference 


Including  set  backs  in  our  definition  of  attrition,  we  find  that, 
controlling  for  enrollees’  demographic  and  quality  characteristics,  enrollment 
in  TNCC  is  associated  with  a  lower  attrition  rate.  Using  regression  results  to 
predict  attrition  associated  with  sending  the  pooled  population  to  each  of  the 
programs,  we  estimate  the  attrition  rate  to  be  nearly  3  times  as  high  for  NSHS 
SD  as  for  TNCC  (29%  academic  attrition  vs.  10%). 1 

There  are  problems  with  including  set  backs  in  our  model.  By  doing  so, 
we  are  counting  certain  enrollees  at  San  Diego  twice — once  for  their  initial 
enrollment  in  an  MLT  course,  then  as  a  second  observation  when  they  are 
dropped  from  that  course  (as  a  set  back)  and  are  enrolled  in  the  next  available 
MLT  course.  To  accurately  deal  with  this  phenomenon,  one  would  want  to 
control  for  the  simultaneity  associated  with  the  decision  to  set  back  a  student 
and  the  final  outcome  of  that  student  in  the  MLT  program  (did  he/she 
ultimately  pass  or  fail  the  course?).  This  can  be  done  using  a  bivariate  probit 
model.  Unfortunately,  because  of  the  limitations  of  our  data  (in  terms  of 
observations,  lack  of  set  backs  at  TNCC,  and  limited  variables  necessary  for 
model  identification),  bivariate  probit  analysis  was  not  feasible. 

An  alternative  is  to  eliminate  set  backs  from  our  definition  of  academic 
attrition  and  eliminate  these  observations  from  the  population  of  interest.  Thus, 
(continued) 


1 .  If  we  were  to  consider  only  those  nonacademic  attrites  classified  as  medical  or 
family/personal  emergency  to  be  random  (treating  motivational,  attitude,  or  misconduct  attrites 
as  nonrandom  events),  the  estimated  attrition  rates  for  the  pooled  population  at  NSHS  SD  and 
TNCC  are  32%  vs.  11%,  respectively. 
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Comparison  of  Academic  Attrition  Rates: 

Results 


•  Attrition  ( including  set  backs  for  SD) 

-  Predicted  rate  for  pooled  population 

TNCC  10%  NSHSSD  29%  (statistically  sig.) 

•  Attrition  ( excluding  set  backs  for  SD  (n=40)) 

-  TNCC  (n=146) 

-  NSHS  SD  (n=144) 

-  Comparing  characteristics  across  two  classes,  GPA 
differential  disappears 

-  Controlling  for  individual  characteristics,  we  find  no 
statistically  significant  difference 


(continued) 

the  population  becomes  the  unique  number  of  students  in  each  of  the  programs 
(minus  nonacademic  attrition).1  Given  this  definition,  we  find  a  total  of  290 
unique  enrollees:  146  at  TNCC  (unchanged  because  they  do  not  have  set 
backs)  and  144  at  NSHS  SD.  And,  the  programs  academically  attrited  (or 
disenrolled)  10  and  22  enrollees,  respectively. 

Comparison  of  the  individual  characteristics  of  this  population  of 
unique  enrollees  by  program  indicates  that  NSHS  SD  has  more  junior 
paygrade  enrollees  who  were  less  likely  to  have  met  prerequisites,  and  fewer 
women,  but  its  enrollee  population  no  longer  has  statistically  lower  Corps 
School  GPAs. 

Using  this  definition  of  academic  attrition,  excluding  set  backs,  we  find 
no  statistically  significant  difference  in  academic  attrition  rates  between  the 
two  MLT  programs  (controlling  for  demographic  and  student  quality 
characteristics).2 

Although  this  alternative  of  excluding  set  backs  allows  us  to  deal  with 
limitations  in  the  data  and  pilot  design,  it  does  overlook  the  costly  impact  of 
using  the  set-back  concept,  especially  using  it  to  such  a  large  extent.  We  will 
look  more  closely  at  the  impact  of  set  backs  in  the  next  slide. 


1 .  The  measure  know  as  "unique  number  of  students”  counts  set  backs  only  once,  even  though 
the  set-back  student  was  ultimately  enrolled  in  two  separate  MLT  courses. 

2.  Treating  motivational,  attitude,  and  misconduct  attrites  as  nonrandom  events,  we  still  find 
no  significant  difference  in  attrition  (excluding  SBs). 
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Exploring  the  Set-Back  Concept: 
Impact  on  Production 

•  Significantly  reduces  the  throughput 

-  202  NSHS  MLT  course  enrollees,  but  159  unique 
individuals 

•  Probability  of  graduating  is  lower  for  set  backs  than 
all  other  SD  enrollees 

-  60%  vs.  96%  (not  controlling  for  population  differences) 

-  If  graduate,  probability  of  passing  ASCP  is  lower 

•  In  this  case,  using  the  set-back  concept  may  not  have 
had  any  impact  on  the  number  of  graduates 


The  typical  observer  would  look  at  NSHS  statistics  for  the  duration  of 
the  pilot  and  conclude  that  the  program’s  throughput  was  202  enrollees 
(compared  to  TNCC's  150).  Although  NSHS  certainly  had  the  capacity  and 
ability  to  enroll  202  students  in  its  MLT  program,  its  true  throughput  was  only 
159  unique  students.  This  is  because  over  one-fourth  of  its  enrollee  population 
(43  of  202)  was  set  back,  or  given  an  opportunity  to  repeat  the  course. 

How  successful  is  the  set-back  concept?  Excluding  nonacademic 
disenrollments,  if  we  separate  NSHS  San  Diego’s  unique  enrollee  population 
into  those  who  were  set  back  and  reenrolled  in  a  class,  and  those  who  were 
never  set  back  (43  and  101,  respectively),  we  find  that  the  probability  of  set 
backs  eventually  graduating  from  the  SD  program  is  significantly  lower  than 
for  those  not  set  back.  Of  those  who  do  graduate,  their  probability  of  passing 
the  ASCP  is  lower  than  for  non- set-back  graduates  (16%  vs.  38%). 

If  NSHS  SD  had  not  used  the  set-back  concept,  instead  disenrolling 
these  43  students  and  admitting  43  new  enrollees,  we  could  expect  this  new 
population  of  43  students  to  have  a  graduation  rate  of  61%  vs.  60%  (assuming 
that,  on  average,  the  new  enrollees  would  have  the  same  characteristics  as  the 
original  159  enrollees  and  attrition  rates  were  constant).1  In  this  example,  this 
leads  to  an  increase  of  .22  of  a  graduate,  or  basically  no  difference. 


1.  Of  the  159  enrollees,  15  were  nonacademic  disenrollments  (9%).  This  leaves  144  enrollees, 
of  which  123  graduated  (97  of  the  101  not  set  back,  26  of  the  43  set  back).  If  the  43  set  backs 
had  been  disenrolled,  the  average  graduation  rate  would  have  been  97/144  =  67%.  Applying 
this  graduation  rate  to  a  new  set  of  43  enrollees  (minus  average  nonacademic  attrites)  yields 
[43  *  ( 1  -.09)]  *  .67  =  26.22,  which  is  a  negligible  difference. 
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Exploring  the  Set-Back  Concept: 
Impact  on  Cost 

•  Three  categories  of  set  backs 

-  Academic  short-term  (within  first  3  months) 

-  Academic  long-term  (sometime  after  first  3  months) 

-  Nonacademic  (personal,  medical,  administrative) 

•  Each  increases  cost  of  training  (adds  minimum  of  3 
months  to  total  training  time) 

-  Whether  a  set  back  eventually  completes  course 
(graduates)  or  fails  course 

-  100  days  of  training  time  translates  to  $10,740  in 
programming  rate 

-  Additional  programming  rate  cost  for  43  set  backs 
=  $461,808  (or  $3,785  per  graduate) 


Even  though  there  does  not  appear  to  be  a  difference  in  the  total 
number  of  graduates  regardless  of  whether  SD  used  the  set-back  concept,  there 
is  a  cost  to  doing  so. 

It  appears  that  set  backs  fall  into  three  categories:  (1)  students  who 
have  trouble  from  the  beginning  and  are  almost  immediately  set  back,  (2) 
those  who  may  successfully  complete  a  portion  of  the  course  but  may  need  to 
repeat  a  section  of  material  that  is  covered  later  in  the  year,  and  (3) 
nonacademic  set  backs  (for  personal,  medical,  or  administrative  reasons).  In 
most  of  these  cases,  setting  the  student  back  adds  3  to  4  months  to  the  total 
training  time  because  they  typically  start  matriculating  in  the  next  available 
class  (classes  start  every  3  to  4  months).1  Considering  a  programming  rate  of 
$39,200  for  each  enlisted  training  billet,  3  months  of  additional  training  time 
adds  approximately  $10,740  to  the  total  training  cost  for  the  student  (whether 
or  not  he/she  ultimately  graduates  from  the  course).2 

Therefore,  using  the  set-back  concept  may  not  have  negatively  affected 
NSHS  SD’s  MLT  outcome  (in  terms  of  total  graduates  for  the  pilot  phase),  but 
it  did  increase  student  training  costs  by  an  estimated  43*$10,740  =  $461,808 
(or  approximately  $3,785  per  graduate). 


1 .  In  one  case,  a  student  was  set  back  for  6  months.  Based  on  NITRAS  data,  we  found  that  the 
average  NSHS  SD  set  back  (during  the  pilot  study)  increased  training  time  by  100  days. 

2.  For  programming  rate  (the  per-enlisted-billet  dollar  amount  that  DHP  pays  to  the  Navy  in 
the  year  of  execution),  we  used  an  average  of  2000  and  2001  enlisted  programming  rates 
(including  PCS,  which  DHP  also  pays  per  billet).  Programming  rate  was  provided  by  N931, 
based  on  1998  and  1999  POMs,  respectively. 
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Exploring  the  Set-Back  Concept: 
More  Limited  Application 

•  Of  NSHS  SD’s  43  set  backs 

-  38  were  academic  short-term  (graduate  rate  58%) 

-  4  were  academic  long-term  (graduate  rate  100%) 

-  1  was  nonacademic  (graduate  rate  100%) 

•  If  NSHS  SD  used  the  set-back  concept  for 
academic  long-term  and  nonacademic  only 

-  Unique  enrollment  would  be  197  vs.  159 

-  Estimated  graduates  =127  (assumes  consistency  in 
population  characteristics  and  attrition  rates) 

-  Additional  programming  rate  cost  for  5  set  backs 
=  $53,698  (or  $423  per  graduate) 


What  if  SD  had  used  set  backs  more  judiciously?  In  the  previous  slide, 
we  described  three  categories  of  set  backs:  short-term  academic,  long-term 
academic,  and  nonacademic.  Looking  at  each  of  these,  we  find  that  NSHS  SD 
had  38,  4,  and  1,  respectively.  These  are  small  numbers,  but  the  graduation 
rates  associated  with  each  category  show  that  long-term  academic  and 
nonacademic  set  backs  appear  to  be  more  successful  than  short-term  academic 
set  backs.  This  is  not  surprising  because  short-term  academic  set  backs  (those 
set  back  within  the  first  3  months  of  the  course)  are  less  likely  to  have  shown 
any  academic  success,  whereas  long-term  academic  and  nonacademic  set 
backs  have  probably  indicated  some  academic  success  or  they  would  have 
been  set  back  earlier. 

So,  what  if  NSHS  SD  had  used  a  more  limited  set-back  concept,  say 
only  for  late  set  backs  (those  who  had  been  at  least  partially  successful)  and 
nonacademic  set  backs?  Again,  assuming  consistency  in  the  student  population 
and  constant  attrition  rates,  we  estimate  that  this  could  have  increased 
graduates  from  123  to  127.1  Under  this  scenario,  SD  set  backs  would  have 
increased  costs  by  only  $53,698  (or  approximately  $423  per  graduate).2 


1 .  If  the  38  short-term  academic  set  backs  had  been  disenrolled,  the  average  graduation  rate 
would  have  been  102/144  =  71%.  Applying  this  graduation  rate  to  a  new  set  of  38  enrollees 
(minus  average  nonacademic  attrites)  yields  [38  *  (1  -.09)]  *  .71  =  24.55.  Adding  this  to  the 
102  graduates  yields  126.55  total  graduates. 

2.  Cost  is  calculated  as  5  set  backs  multiplied  by  0.274  (or  100  days  /  365)  of  the  2000-2001 
average  enlisted  program  rate  or  $10,740  per  SB  (0.0274  x  $39,200).  This  total  cost  of 
$53,698  is  divided  by  127  graduates  to  obtain  the  per-graduate  increase  ($53,698  /  127  = 
$423). 
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Cost  Efficiency 


•  Is  TNCC  more  cost  effective  than  alternative 
Navy  MLT  training  programs? 

•  In  the  short  term  (excluding  student  salaries) 

-  Compare  total  per-graduate  cost  to  NSHS  San  Diego 

-  Account  for  attrition  and  set  backs 

-  Include  cost  of  Portsmouth-provided  overhead  and 
resources 

•  Recommend  long-term  follow-up 

-  Evaluate  life-cycle  cost  (impact  on  retention) 


Next,  we  turn  to  cost  efficiency.  Navy  Medicine  is  continuously  under 
pressure  to  control  costs.  In  fact,  cost  containment/reduction  was  the  initial 
motivation  for  looking  at  outsourcing  options.  Therefore,  we  must  ask  whether 
outsourcing  MLT  training  leads  to  higher  or  lower  overall  training  cost. 

To  answer  this  question  in  the  short  term,  we  compared  the  total  per- 
graduate  cost  of  TNCC  and  NSHS  San  Diego  MLT  programs,  excluding 
student  salaries.  By  looking  at  cost  per  graduate  rather  than  total  program  costs 
or  cost  per  student,  we  account  for  the  additional  costs  associated  with 
attrition.  We  also  include  NSHS  Portsmouth’s  provided  overhead  and  resource 
costs  (base  operating  support,  academic  and  administrative  overhead, 
instructors,  and  transportation)  to  the  TNCC  program. 

While  the  pilot  time  frame  did  not  allow  for  it,  we  recommend 
evaluating  cost  efficiency  over  the  long  term  because  there  may  be  long-run 
cost  implications  of  the  TNCC  program.  Keep  in  mind  that  the  amortized  cost 
of  Navy  training  goes  up  or  down  depending  on  how  many  years  a  trained 
MLT  remains  in  the  Navy.  If  TNCC’s  MLT  graduates  tend  to  have  lower 
retention  than  NSHS  MLT  graduates,  the  Navy  has  fewer  years  to  recoup  its 
training  investment,  increasing  the  life-cycle  cost  for  TNCC  grads.  Navy 
Medicine  should  commit  to  following  graduates  from  this  pilot  program 
throughout  their  careers,  at  least  through  their  first  retention  decision  point. 
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Method  for  Comparing  Short-Run  Costs: 
NSHS  San  Diego  and  TNCC 

•  Using  predicted  attrition  rates  (controlling  for 
enrollee  characteristics),  we  estimate  the  number  of 
graduates  for  each  program  to  be: 

(enrollees  -  nonacacl.  attrites)  *  (1  -  predicted  acad.  attrition  rate) 

•  Actual  cost  data  were  collected  from  NSHS  PTS, 

NSHS  SD,  and  MED-51 

-  Course-specific  (AD  instructors,  classroom  expenses, 
tuition  and  incentive  fee,  transportation,  TAD/training) 

-  Activity- specific  (academic  and  administrative  support, 
G&A,  BOS) 


When  comparing  the  short-run  cost  per  graduate  across  programs,  we 
estimated  the  number  of  graduates  for  each  program  using  the  predicted 
attrition  rates.  This  methodology  controls  for  differences  in  enrollee 
characteristics  (demographics  and  student  quality)  that  may  have  existed 
between  the  two  programs’  enrollee  populations. 

To  estimate  graduates,  we  start  with  the  actual  number  of  enrollees  in 
each  program  (for  the  total  duration  of  the  pilot  time  frame).  We  exclude  non- 
academic  attrites  from  each  program’s  enrollee  population.1  Then  using  each 
program’s  predicted  academic  attrition  rate,  we  estimated  the  total  number  of 
graduates  for  NSHS  San  Diego  and  TNCC.2 

For  program  cost,  we  collect  actual  cost  data  from  each  of  the 
schoolhouses  and  MED-51.  The  cost  data  include  MLT  course- specific  costs 
(such  as  AD  course  instructors,  books,  lab  supplies,  tuition,  transportation)  and 
activity,  or  schoolhouse-specific,  costs  (such  as  NSHS  overhead).  For  more 
details  on  the  cost  calculations,  see  appendix  E. 


1 .  Nonacademic  attrites  are  considered  to  be  random  (nonacademic  attrites  for  NSHS  SD  were 
six  medical  and  nine  other  (one  personal  and  eight  attitude/misconduct);  for  TNCC,  three 
medical  and  one  other  (attitude/misconduct)).  See  slide  on  page  23. 

2.  Recall  that  the  predicted  academic  attrition  rate  for  each  program  is  calculated  using  the 
regression  results  from  the  attrition  model  discussed  previously.  These  rates  represent  the 
mean  predicted  outcome  associated  with  sending  the  pooled  population  to  TNCC  vs.  sending 
exactly  the  same  population  to  NSHS  San  Diego. 
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Comparison  of  Short-Run  Costs: 
NSHS  San  Diego  and  TNCC 

•  Estimate  that  TNCC  cost  per  graduate  is  6%  less 
than  NSHS  SD 

-  Controlling  for  enrollee  demographic  and  student  quality 

-  Excluding  student  salary  (set  backs  are  more  costly) 

-  Including  both  course  and  activity  specific  costs 

•  15%  less  for  course- specific  costs 

-  $10,285  vs.  $12,126 

•  5%  more  for  activity -specific  (NSHS  support)  cost 

-  $11,361  vs.  $10,858 

-  Contracting  institution  has  no  control 

•  Total  TNCC  program  savings  (for  estimated  132 
graduates)  is  $176,500  to  $243,800 


Our  estimates  indicate  that  the  per-graduate  cost  for  TNCC  is  6%  less 
than  the  per-graduate  cost  for  the  Navy’s  in-house  MLT  program.  Again,  our 
estimates  control  for  attrition  differences  between  the  programs  due  to 
variation  in  enrollee  demographics  and  student  quality.  Also,  recall  that  these 
estimates  do  not  include  student  salary,  the  most  expensive  portion  of  training 
costs  for  the  MLT  program.1  Based  on  NSHS  SD’s  extensive  use  of  set  back, 
and  how  set  backs  affect  training  cost  in  terms  of  increased  student  salary,  we 
would  expect  to  see  this  difference  in  training  cost  increase  significantly. 

Finally,  these  estimates  include  both  course-specific  costs  (those 
associated  only  with  the  MLT  course,  such  as  MLT  instructors,  tuition,  books, 
transportation)  as  well  as  activity- specific  costs  (those  associated  with  the 
naval  schoolhouses’  administration  and  overhead).  To  adequately  evaluate  the 
TNCC  program,  we  think  it  is  important  to  separate  these  two  types  of  costs 
because  the  contractor  has  no  control  over  NSHS  overhead. 

In  fact,  when  we  look  at  differences  in  course-specific  costs  only, 
TNCC’s  per-graduate  cost  advantage  rises  from  6%  to  15%  over  NSHS  San 
Diego’s  MLT  program.  This  is  because  NSHS  San  Diego  has  a  cost  advantage 
over  NSHS  Portsmouth  when  it  comes  to  activity- specific  costs. 


1 .  For  example,  the  total  cost  to  train  an  MLT  at  NSHS  SD  (excluding  student  salary)  is 
estimated  to  be  $23,162.  For  this  program  (which  runs  369  days),  the  student  salary  in  terms  of 
the  programming  rate  is  about  $39,630  (that  is  $39,000  *  (369/365)),  assuming  a  100% 
graduation  rate.  The  student  salary  per  graduate  increases  as  (a)  the  rate  of  attrition  increases, 
and  (b)  as  we  showed  earlier,  as  the  number  of  set  backs  increases. 
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Student  Satisfaction 


•  Examine  and  compare  student’s  satisfaction  with: 

-  Individual  courses  and  the  program  as  a  whole  (TNCC  and 
NSHS  San  Diego) 

-  Quality  of  life  (TNCC  and  NSHS  Portsmouth) 

•  Measure  satisfaction  via  standardized  surveys  given  at 
TNCC  and  the  appropriate  control  site 

•  Conduct  post-graduation  survey 

-  Did  program  adequately  prepare  you  for  your  current  job? 
(TNCC  and  NSHS  San  Diego  graduates) 


In  addition  to  cost  and  the  ability  to  produce  quality  graduates,  BUMED 
is  determined  that  a  successful  program  should  not  lead  to  lower  student 
satisfaction  with  the  individual  courses,  the  program  as  a  whole,  or  quality  of 
life  while  attending  C-school.  Although  the  program’s  failure  in  any  of  these 
areas  would  not  signal  an  overall  failure  of  the  TNCC  pilot,  it  would  indicate 
areas  for  concentration  and  improvement  in  the  future. 

For  our  evaluation,  student  satisfaction  was  measured  at  TNCC  and 
control  sites  using  several  surveys.  Throughout  the  program,  students  were 
asked  to  evaluate  their  courses  (didactic  and  clinical),  their  instructors,  access 
to  instructors,  and  other  quality-of-life  (QOL)  issues  (transportation,  sick  call, 
PRT,  messing,  housing/berthing,  and  military  support  and  requirements 
available  for  advancement,  career  counseling,  leave,  etc.). 

The  program’s  graduates  have  also  had  a  chance  to  express  their  level 
of  satisfaction  with  the  training  program  once  they  have  had  the  opportunity  to 
apply  their  new  skills  and  knowledge  at  their  first  duty  station.  The  post¬ 
graduation  survey  (given  4  and  12  months  after  graduation)  asks  graduates  if 
they  felt  as  though  the  training  program  adequately  prepared  them  for  working 
in  the  lab  (thorough  knowledge  of  theory  and  technical  applications). 

As  mentioned  before,  TNCC  students’  satisfaction  with  the 
courses/program  was  compared  with  MLT  students’  satisfaction  at  NSHS  San 
Diego.  TNCC  students’  satisfaction  with  QOL  was  compared  with  the 
satisfaction  of  students  attending  other  courses  of  similar  duration  at  NSHS 
Portsmouth  (advanced  X-ray  and  Pharmacy  Technician  C-schools). 
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Satisfaction  with  MLT  Program: 
NSHS  San  Diego  and  TNCC 

•  Population  is  MLT  graduates  who  completed  end-of- 
course  survey 

-  TNCC  (n=125)  NSHS  SD  (n=109) 

•  Characteristics  of  students  completing  the  courses  are 
very  similar 

-  NSHS  SD  grads  are  more  junior  and  less  likely  to  be  female 

•  Students  were  asked,  “how  satisfied  are  you  with. . 

-  Textbooks,  equipment,  computer  and  library  resources,  etc. 

-  Instructors  and  program  oversight 

-  Classroom  environment 

-  Program’s  assistance  in  preparing  you  for  clinical  rotations, 
to  be  an  MLT,  to  take  registry  exam,  to  get  a  degree 


We  begin  by  looking  at  the  students’  satisfaction  with  the  MLT 
programs.  The  population  of  interest  is  all  Navy  and  TNCC  graduates  who 
completed  end-of-course  surveys  (within  the  pilot  time  frame).  Of  the  258  total 
graduates,  234  responded  to  the  survey.1 

Student  quality  and  demographic  characteristics  for  graduates  from  the 
two  programs  were  fairly  similar,  with  the  exception  of  paygrade  and  gender. 
NSHS  San  Diego  graduates  were  more  likely  to  have  started  the  program  as  an 
E-l  or  E-2  (47%  vs.  29%)  and  less  likely  to  be  female  (27%  vs.  40%). 

In  the  end-of-course  survey,  students  were  asked  to  rank  their 
satisfaction  level  with  several  aspects  of  the  program  resources,  the  instructors 
and  instructor  oversight,  the  classroom  environment,  and  the  program’s 
assistance  in  preparing  the  student  for  clinical  rotations,  to  take  the  ASCP 
exam  to  get  a  degree,  and  ultimately  to  be  an  MLT.2 

Controlling  for  population  characteristics,  we  compared  whether  a 
student  indicated  that  he/she  was  satisfied. 


1 .  The  overall  response  rate  for  the  end-of-course  survey  (administered  by  the  respective 
MLT  programs)  was  91%  (slightly  higher  for  TNCC  than  NSHS  SD — 92%  vs.  89%). 

2.  Students  were  asked  to  rank  their  satisfaction  on  a  5  -point  scale  ( l=very  satisfied, 
2=somewhat  satisfied,  3=neither  satisfied  or  dissatisfied,  4=somewhat  dissatisfied,  5= very 
dissatisfied).  We  collapsed  responses  into  a  dichotomous  indicator  variable.  This  variable  was 
set  equal  to  1  to  indicate  a  response  of  somewhat  or  very  satisfied,  or  to  0  to  indicate  a 
response  of  neither  satisfied  nor  dissatisfied,  to  very  dissatisfied. 
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Satisfaction  with  MLT  Program 

(Statistically  Significant  Differences) 


Percent  satisfied 


Satisfaction  with 

TNCC 

NSHS  SD 

-  Classroom  instruction 

92% 

77% 

-  Instructors’  ability 

89% 

78% 

-  Instructors’  preparation 

91% 

67% 

-  Monitoring  of  instructors 

83% 

61% 

-  Preparation  for  clinical 

82% 

65% 

-  Preparation  for  ASCP  exam 

72% 

53% 

-  MLT  program  overall 

91% 

62% 

For  7  of  the  16  survey  questions  dealing  with  various  aspects  of  the 
MLT  programs,  we  found  that  attendance  at  TNCC  was  associated  with  higher 
rates  of  satisfaction  among  graduates  (statistically  significant  controlling  for 
student  quality  and  demographics).  These  7  areas  are  listed  above,  as  are  the 
predicted  satisfaction  rates  associated  with  sending  the  pooled  MLT  graduate 
population  to  TNCC  versus  NSHS  San  Diego.1 

Many  of  these  areas  deal  with  the  quality  of  instruction,  which  is  not 
surprising  because  this  is  an  area  in  which  we  might  expect  the  community 
college  to  have  an  advantage.  While  the  in-house  programs  rely  on  fully 
trained  Navy  MLTs  to  teach  the  course,  the  college  has  access  to  professors 
that  specialize  in  each  of  the  MLT  course  modules  (biologist,  chemist, 
mathematician,  etc.).  One  might  expect  this  breadth  of  expertise  to  lead  to  a 
more  comprehensive  and/or  stimulating  learning  environment. 


1 .  See  appendix  F  for  student  satisfaction  with  program  comparison  population  means  and 
regression  results. 
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Satisfaction  with  MLT  Program 

(No  Statistically  Significant  Differences) 

Satisfaction  with 

Percent  satisfied 

TNCC  NSHS  SD 

-  Classroom  discipline 

no  difference 

-  Fellow  student’s  interest 

no  difference 

-  Access  to  instructors 

no  difference 

-  Instruction  of  clinical  rotation 

no  difference 

-  Quality  of  equipment 

no  difference 

-  Quality  of  texts 

no  difference 

-  Access  to  computer  lab 

no  difference 

-  Prep  to  be  entry-level  MLT 

no  difference 

-  Prep  to  receive  associate  degree 

no  difference 

Although  we  found  no  cases  in  which  satisfaction  with  the  program 
was  statistically  significantly  lower  for  TNCC  graduates,  we  felt  it  was 
important  to  look  at  those  aspects  of  the  program  for  which  there  was  no 
estimated  difference  in  satisfaction  among  the  programs’  graduates. 

On  one  hand,  consider  classroom  discipline  and  access  to  instructors. 
Because  the  in-house  programs  are  considered  to  be  so  strong  in  these  areas, 
there  was  a  great  deal  of  concern  whether  outsourced  training  could  match 
them.  At  least  from  the  graduate’s  perspective,  there  appeared  to  be  little 
difference  between  the  programs,  or  at  least  not  enough  to  affect  satisfaction.1 

On  the  other  hand,  one  might  have  expected  TNCC  to  have  been  better 
at  preparing  the  graduate  to  receive  an  associate  degree.  Again,  from  the 
graduate’s  perspective,  there  was  no  difference  between  the  programs. 


1.  Only  graduates  were  surveyed.  One  might  conjecture  that,  if  all  enrollees  had  been 
surveyed,  the  findings  might  be  different  (for  example,  if  all  those  for  whom  discipline  and 
lack  of  access  to  instructors  was  an  issue  failed  to  successfully  complete  the  program  and 
therefore  were  not  surveyed).  But,  recall  that  attrition  rates  at  TNCC  were  either  less  than  or 
equal  to  NSHS  San  Diego’s  (depending  on  the  measure  used). 
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Student  Satisfaction  with  MLT 
Program:  Post-Graduation 

•  Would  graduates’  impression  of  the  program 
change  once  they  had  the  opportunity  to  apply 
what  they  had  learned? 

•  MLT  graduates  who  completed  4-month  post¬ 
graduation  survey 

-  TNCC  (n=130)  NSHS  SD  (n=109) 

•  Characteristics  are  very  similar  among  graduates 

-  NSHS  SD  had  more  graduates  who  started  the  program 
as  E-l  to  E-3  (60%  vs.  28%)  and  fewer  females  (23% 
vs.  38%) 


Whereas  the  end-of-course  survey  looked  at  student  satisfaction  with 
the  program  at  the  point  of  graduation,  the  post-graduation  survey  was 
designed  to  measure  a  graduate’s  satisfaction  with  how  well  the  MLT  program 
prepared  him/her  with  the  knowledge  of  theory  and  principles  and  the  ability 
to  apply  that  knowledge  (technical  skill  in  the  lab)  as  an  MLT  at  his/her  first 
duty  station. 

The  surveys  were  conducted  at  two  points  in  time,  4  months  and  12 
months  after  graduation.  The  surveys  were  administered  by  telephone  to  Navy 
graduates  from  both  TNCC  and  NSHS  San  Diego.1  To  date,  we  have 
completed  the  4-month  survey  for  all  MLT  graduates.  The  response  rate  for 
this  survey  was  93%  (96%  for  TNCC  and  89%  for  NSHS  San  Diego 
graduates).  The  12-month  post- graduation  surveys  will  not  be  completed  until 
September  2002,  but  we  do  have  interim  results  based  on  completed  12-month 
surveys  for  8  of  the  10  MLT  classes  in  the  pilot. 

Locusing  on  graduates  who  responded  to  the  4-month  survey,  we  again 
see  that  the  only  characteristics  for  which  we  find  statistically  significant 
differences  between  the  two  populations  (TNCC  and  NSHS  San  Diego)  are 
gender  (TNCC  has  more  female  graduates)  and  paygrade  (TNCC  has  fewer 
graduates  who  began  the  program  as  E-ls,  E-2s,  or  E-3s).2 


1.  Historically,  NSHS  San  Diego  and  NSHS  Bethesda  surveyed  MLT  graduates  at  their  first 
duty  station.  These  mail -out  surveys  yielded  a  poor  response  rate.  To  maximize  participation 
during  the  pilot,  we  recommended  conducting  the  post -graduation  surveys  (both  a  graduate 
and  supervisor  version)  via  telephone. 

2.  See  appendix  H  for  population  means  and  regression  results  associated  with  the  post¬ 
graduation  graduate  survey  .  o7 


Student  Satisfaction 


Comparing  Satisfaction  with 
Program  Post-Graduation 

•  At  4  months  after  graduation,  we  find  no 
statistically  significant  difference  in  graduates’ 
satisfaction  with  program  prep  for  current  job 

•  Interim  findings  for  12  months  after  graduation 
indicate 

-  Higher  satisfaction  levels  associated  with  graduation 
from  TNCC  (statistically  significant  differences  for 
predicted  satisfaction  rates  for  pooled  population) 

•  Knowledge  of  theory -TNCC  68%  NSHS  SD  50% 

•  Technical  application- TNCC  71%  NSHS  SD  53% 


At  4  months  after  graduation,  we  find  no  statistically  significant 
differences  in  the  graduates’  satisfaction  with  the  program  (in  terms  of 
providing  the  knowledge  of  theory  and  principles  and  the  technical  skill 
required  to  work  in  the  lab). 

Looking  at  interim  findings  for  the  12-month  post-graduation  survey, 
we  do  find  evidence  of  higher  rates  of  satisfaction  associated  with  graduation 
from  TNCC.  Shown  above  are  the  predicted  satisfaction  rates  associated  with 
having  sent  the  pooled  sample  of  survey  respondents  to  TNCC  versus  sending 
them  to  NSHS  SD. 
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Student  Satisfaction 


Satisfaction  with  QOL: 
NSHS  Portsmouth  and  TNCC 


•  Population  is: 

-  TNCC  MLT  graduates  who  completed  the  end-of-didactic 
phase  survey  (n=136) 

-  NSHS  Portsmouth  X-ray  and  Pharmacy  Tech  graduates  who 
completed  end-of-course  surveys  (n=261) 

•  TNCC  graduates  were  more  likely 

-  To  be  female  (39%  vs.  29%) 

-  To  live  off  base  (56%  vs.  41%) 

-  Otherwise  demographically  similar  to  X-ray  and  Pharmacy 

•  We  asked, “how  satisfied  are  you  with. . 

-  Access  to  services,  logistics,  support  to  meet  military  and 
personal  requirements 


We  now  shift  to  the  students’  satisfaction  with  quality  of  life  (QOL).  It 
was  determined  early  on  that,  in  terms  of  QOL,  the  goal  for  the  pilot  program 
was  that  QOL  satisfaction  be  no  less  than  for  Navy  students  attending  training  at 
NSHS  Portsmouth.  Therefore,  the  control  group  for  QOL  was  graduates  from 
NSHS  Portsmouth’s  X-ray  Technician  and  Pharmacy  Technician  C-schools. 

The  specific  concern  with  regard  to  QOL  was  this:  during  the  time  that 
Navy  MLT  students  were  attending  courses  on  the  TNCC  campus  (the  6-month 
didactic  phase),  would  they  be  able  to  maintain  the  same  QOL  standards  as 
those  students  who  attended  courses  on  base  at  the  NSHS  Portsmouth?  TNCC 
students  were  surveyed  at  the  end  of  their  didactic  phase.  Because  the  control 
group  was  surveyed  only  at  the  end  of  the  course  (capturing  graduates  only),  we 
limited  the  TNCC  sample  to  graduates.1 

Population  characteristics  for  the  QOL  analysis  were  limited  to 
demographics  (gender,  paygrade,  marital  status,  children,  whether  family 
accompanied  student  on  training  tour,  and  whether  one  lived  on  or  off  base). 
Compared  to  NSHS  Portsmouth  graduates,  TNCC  graduates  were  more  likely  to 
be  female  and  to  have  lived  off  base. 

The  QOL  portion  of  the  survey  asked  students  to  rank  their  satisfaction 
with  access  to  services,  logistics,  and  support  necessary  to  meet  both  military 
and  personal  requirements  while  a  student.2 


1 .  Fourteen  TNCC  surveys  were  deleted  from  the  QOL  analysis  because  of  this  criterion. 

2.  Again,  satisfaction  rankings  were  based  on  a  5-point  scale  and  collapsed  to  a  dichotomous 
variable  (l=somewhat  to  very  satisfied,  0=neither  satisfied  or  dissatisfied  to  very  dissatisfied). 
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Student  Satisfaction 


Satisfaction  with  QOL 

(Statistically  Significant  Differences) 


Percent  satisfied 


Satisfaction  with 

TNCC 

NSHS  PTS 

-  Your  Welcome  Aboard  packet 

63% 

39% 

-  Student  lounge 

73% 

56% 

-  Your  locker 

89% 

80% 

-  Access  to  adequate  meals 

74% 

48% 

-  Group  PT  sessions 

58% 

39% 

-  Quantity  and  quality  of  study  sessions 

40% 

75% 

-  Time  to  handle  personal  matters 

55% 

70% 

-  Uniform  and  civilian  attire  policy 

62% 

77% 

-  Opportunity  to  pursue  advancement 

39% 

58% 

Of  the  15  QOL  questions  asked  on  the  survey,  we  found  statistically 
significant  differences  in  the  rate  of  satisfaction  for  9  QOL  areas  (controlling 
for  demographic  characteristics).  For  some,  attendance  at  TNCC  was 
associated  with  higher  rates  of  satisfaction;  for  others,  TNCC  was  associated 
with  lower  satisfaction  rates.  The  predicted  rates  of  satisfaction  associated  with 
having  sent  the  pooled  sample  to  TNCC  versus  sending  them  to  NSHS  SD  are 
shown  above.1 

Though  it  is  not  surprising  or  concerning  that  TNCC  students  might  be 
more  dissatisfied  with  the  civilian  attire  policy  (given  that  they  attend  classes 
on  a  civilian  campus),  what  is  more  concerning  is  lower  satisfaction  with 
regard  to: 

•  Time  to  handle  personal  matters,  and 

•  Opportunity  to  pursue  advancement. 

Time  is  an  issue  for  the  TNCC  MLT  students  because  the  program  is 
very  challenging  and  time  intensive,  and  they  do  spend  most  of  their  first  6 
months  away  from  the  base  and  base  resources.  These  measures  should  be 
watched  for  future  rounds  of  TNCC  students,  and  TNCC  graduates  from  the 
pilot  phase  should  be  followed  to  see  if  there  is  any  negative  impact  on  their 
rate  of  advancement. 


1 .  See  appendix  G  for  student  QOL  satisfaction  comparison  population  means  and  regression 
results. 
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Student  Satisfaction 


Satisfaction  with  QOL 

(No  Statistically  Significant  Differences) 

Satisfaction  with 

Percent  satisfied 

TNCC  NSHSPTS 

-  Commute 

no  difference 

-  Access  to  health  care 

no  difference 

-  Access  to  OTC  medications 

no  difference 

-  Orientation  to  TNCC  or  NSHS 

no  difference 

-  The  handling  of  your  pay,  etc. 

no  difference 

-  Help  overall  in  meeting  your  needs 

no  difference 

As  with  program  satisfaction,  we  felt  it  was  also  important  to  look  at 
those  aspects  of  QOL  for  which  there  was  no  estimated  difference  in 
satisfaction  among  TNCC  and  NSHS  Portsmouth  graduates. 

There  was  no  statistically  significant  difference  in  satisfaction  rates 
with  regard  to  one’s  commute,  access  to  health  care,  access  to  over-the-counter 
(OTC)  medications,  and  handling  of  pay.  Again,  these  were  areas  that  Navy 
Medicine  was  especially  concerned  with,  but  they  do  not  appear  to  have  been  a 
problem  with  outsourcing  training  to  a  civilian  off-base  institution. 
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End-  User  Satisfaction 


End-User  Satisfaction 


•  How  well  do  TNCC  MLT  graduates  perform  in 
their  jobs? 

•  Compare  supervisor  ranking/evaluation  of 
graduates’  performance  to  NSHS  graduates 

-  Technical  skills  and  knowledge 

-  Performance  relative  to  peers 

-  Survey  of  supervisors  of  NSHS  San  Diego  and  TNCC 
graduates  at  first  duty  station  (4  and  12  months  after 
graduation) 


Next,  we  look  at  end-user  satisfaction.  Navy  Medicine  did  not  want  to 
lose  sight  of  one  of  its  major  goals  for  the  training  program — to  provide  high- 
quality  Navy  MLTs  to  its  hospital,  clinic,  and  deployable  platform  laboratories 
(considered  the  end-user  of  the  training  product).  We  have  already  looked  at 
one  measure  of  quality  (i.e.,  pass  rates  on  ASCP).  Another  way  to  measure  the 
quality  of  MLT  graduates  is  to  look  at  the  satisfaction  of  the  end  users  with  the 
program’s  graduates.  We  measured  this  using  a  post-graduation  survey  of  the 
graduates’  immediate  supervisors  for  their  first  MLT  tour  of  duty.  The  survey 
was  designed  to  measure  a  supervisor’s  satisfaction  with  the  graduate’s 
knowledge  of  theory  and  principles  and  his/her  ability  to  apply  that  knowledge 
(technical  skill  in  the  lab).  In  addition,  supervisors  were  asked  to  rank  the 
graduate’s  performance  in  the  lab,  relative  to  his/her  peers.1 

The  survey  was  administered  4  and  12  months  following  graduation. 

The  thought  was  that  the  first  survey  would  measure  how  well  the  program 
prepared  the  graduates  for  working  in  the  lab.  If  there  appeared  to  be  problems 
at  that  point,  the  second  survey  would  help  to  determine  whether  they  were 
resolved  after  a  full  year  on  the  job  or  if  performance  differences  continued  to 
persist. 

The  surveys  were  administered  by  telephone  to  the  immediate  laboratory 
supervisors  of  Navy  graduates  from  both  TNCC  and  NSHS  San  Diego. 


1 .  Supervisors  were  asked  to  rank  the  graduates  on  5  -point  scale  ( l=well  above  satisfactory, 
2=above  satisfactory,  3=satisfactory,  4=below  satisfactory,  5=well  below  satisfactory).  We 
collapsed  responses  into  a  dichotomous  indicator  variable  ( I  =abcve  to  well  above  satisfactory, 
O=satisfactory  to  well  below  satisfactory). 


End-  User  Satisfaction 


Comparing  Supervisor  Satisfaction: 
NSHS  San  Diego  and  TNCC 

•  Population  is  MLT  program  graduates  (with  a 
completed  4-month  post- graduation  supervisor  survey) 

-  TNCC  (n=134) 

-  NSHS  SD  (n=  112) 

•  Characteristics  are  very  similar  among  graduates 

-  TNCC  had  more  female  graduates  (39%  vs.  24%) 

•  Predicted  supervisor  satisfaction  rates  for  pooled 
population  show  no  statistically  significant  differences 
at  4  months  after  graduation 

-  Looking  only  for  above  satisfactory  to  well  above 

•  Still  no  statistically  significant  differences  at  12  months 
after  graduation 


We  focus  primarily  on  the  4-month  post-graduation  survey  of 
supervisors  (because  the  12-month  survey  has  not  been  completed).  We  were 
able  to  successfully  survey  the  supervisors  of  246  out  of  258  graduates.  This 
represents  a  response  rate  of  95%. 

Graduates’  demographic  and  student  quality  characteristics  were  very 
similar  between  the  two  programs,  with  the  exception  of  gender.  Controlling 
for  graduates’  characteristics,  we  found  no  statistically  significant  difference  in 
supervisor  satisfaction  associated  with  TNCC  versus  NSHS  San  Diego.  This 
result  holds  for  interim  results  with  the  12-month  survey  (we  have  completed 
the  12-month  post-graduation  survey  for  8  of  the  10  MLT  classes).1 


1 .  See  appendix  I  for  population  means  and  regression  results  associated  with  the  post¬ 
graduation  supervisor  survey. 
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Military  Bearing 


Military  Bearing 


•  Is  there  a  degradation  of  military  bearing 
among  TNCC  MLT  graduates? 

•  Compare  incidence  of  discipline  problems 
for  NSHS  San  Diego  and  TNCC  MLT 
students 

•  Survey  TNCC  and  NSHS  MLT  graduates 
and  their  supervisors  at  first  duty  station 
-  4  and  12  months  after  graduation 


A  final  area  of  concern  is  that  outsourced  training,  conducted  at  a 
civilian  facility,  may  have  a  deleterious  effect  on  the  militarization  of  the 
active-duty  students  attending  the  program.  In  both  the  planning  and 
implementation  stages  of  the  pilot,  great  effort  was  made  to  design  a  program 
that  would  minimize  any  loss  of  military  bearing  (student  sailors  live  on  base, 
there  is  an  active-duty  Navy  oversight  team  with  offices  on  the  TNCC  campus, 
and  the  clinical  phase  is  taught  at  NMC  Portsmouth). 

For  NSHS  San  Diego  and  TNCC  MLT  students,  we  compared  the 
number  of  disciplinary  actions  that  led  to  the  generation  of  a  Minor  Offense 
Report  (MOR)  or  higher  (e.g.,  Disciplinary  Review  Board,  Executive  Officer's 
inquiry  (XOI),  Mast).  The  MLT  working  group  concluded  that  this  would  be 
the  best  way  to  determine  whether  there  is  a  difference  between  the  two 
programs  in  terms  of  developing  and  maintaining  a  military  bearing  among  the 
students. 

In  addition,  we  surveyed  both  the  graduates  and  their  immediate 
supervisors  regarding  the  graduates’  understanding  of  and  ability  to  adhere  to 
the  appropriate  standards  of  military  behavior,  bearing,  and  discipline  that 
were  expected  at  their  first  post-graduation  duty  site.  These  questions  were 
part  of  the  post-graduation  telephone  surveys  conducted  with  graduates  and 
their  immediate  supervisors  4  and  12  months  following  graduation  from  San 
Diego  and  TNCC. 
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Military  Bearing 


Comparing  Disciplinary  Action  Rates: 
NSHS  San  Diego  and  TNCC 

•  Population  is  unique  MLT  program  enrollees 
(count  set  backs  once) 

-  TNCC  (n=150  enrolled) 

-  NSHS  SD  (n=159  enrolled) 

•  Compared  with  TNCC,  NSHS  SD  enrollees  had: 

-  More  enrollees  missing  prerequisites  (9%  vs.  3%) 

-  Fewer  enrollees  with  previous  college  (5%  vs.  6%) 

-  More  E-l,  E-2,  E-3  enrollees  (60%  vs.  34%) 

-  Fewer  female  enrollees  (20%  vs.  36%) 

•  Predicted  disciplinary  action  rates  for  pooled 
population  show  no  statistically  significant  differences 
between  programs  (only  21  disciplinary  incidences  total) 


Starting  with  the  incidence  of  disciplinary  actions,  we  focus  on  the  set 
of  unique  enrollees  in  either  the  TNCC  or  NSHS  San  Diego  MLT  programs 
(by  unique  enrollees  we  mean  that  San  Diego  set  backs  are  only  counted  once). 

Comparing  demographic  and  student  quality  characteristics  of 
enrollees  between  the  two  programs,  we  find  that  NSHS  SD  enrollees  were 
more  likely  to  be  missing  prerequisites,  more  likely  to  be  in  a  junior  paygrade, 
and  less  likely  to  have  had  previous  college.  San  Diego  also  had  fewer  female 
enrollees. 

Controlling  for  both  demographic  and  student  quality  characteristics, 
we  found  no  statistically  significant  difference  in  the  incidence  of  disciplinary 
actions  for  TNCC  versus  NSHS  San  Diego  enrollees.1 

Note  that  disciplinary  actions  were  rare  in  either  program.  In  total,  out 
of  309  enrollees,  only  21  (or  less  than  7%)  were  written  up  for  some 
disciplinary  infraction. 


1 .  See  appendix  J  for  disciplinary  action  comparison  population  means  and  regression  results. 
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Military  Bearing 


Post-Graduation  Surveys: 
NSHS  San  Diego  and  TNCC 

•  Population  is  MLT  program  graduates  (with  a 
completed  4-month  post-graduate  survey) 

-  Graduates:  TNCC  (n=130)  NSHS  SD  (n=109) 

-  Supervisors:  TNCC  (n=134)  NSHS  SD  (n=112) 

•  Characteristics  are  very  similar  among 
graduates 

-  NSHS  SD  had  more  graduates  who  started  the 
program  as  E- 1  to  E-3  and  fewer  females 


Turning  to  the  post -graduation  surveys,  again  we  focus  on  the 
population  of  graduates  with  a  completed  4-month  survey  (239  graduates  and 
the  supervisors  of  246  graduates  were  successfully  surveyed). 

Again,  graduates  from  both  programs  were  similar  in  terms  of 
demographics  and  student  quality,  with  the  exception  of  gender  and  pay  grade.1 


1 .  See  appendices  H  and  I  for  population  means  and  regression  results  for  the  post-graduation 
graduate  and  supervisor  surveys,  respectively. 
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Military  Bearing 


Comparing  Supervisor  Satisfaction: 
NSHS  San  Diego  and  TNCC 

•  Supervisors  are  asked  if  graduates  are 

-  Able  to  adhere  to  appropriate  standards 
of  military  behavior ,  bearing ,  and 
discipline 

-  Exhibiting  the  proper  attitude  toward 
the  job 

•  Predicted  supervisor  satisfaction  rates  for  pooled 
population  show  no  statistically  significant  differences 

-  Looking  only  for  above  satisfactory  to  well  above 

•  Interim  findings  for  the  12-month  survey  indicate 

-  Lower  satisfaction  with  TNCC  graduate’s  attitude 

TNCC  74%  NSHS  SD  86%  (statistically  significant) 


For  the  supervisor  survey,  the  graduate’s  immediate  supervisor  was 
asked  to  rank  (a)  the  graduate’s  ability  to  adhere  to  the  appropriate  standards 
of  military  behavior,  bearing,  and  discipline  and  (b)  whether  he/she  exhibits 
the  proper  attitude  toward  the  job.  Responses  were  ranked  on  a  5-point  scale 
(well  above  average  to  well  below  average).  We  collapsed  responses  into  a 
dichotomous  indicator  variable  (l=above  average  to  well  above  average, 
0=average  to  well  below  average). 

At  4  months  after  graduation,  controlling  for  graduate  characteristics, 
we  found  no  statistically  significant  difference  with  the  rates  of  supervisor 
satisfaction  in  this  area  (looking  for  above-average  responses).  Interim 
findings  based  on  the  partially  completed  12-month  surveys,  however,  did 
indicate  that  TNCC  was  associated  with  lower  rates  of  supervisor  satisfaction 
with  regard  to  the  graduate’s  attitude  toward  the  job.  We  will  continue  to 
monitor  this  finding  as  we  complete  the  12- month  surveys. 
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Military  Bearing 


Graduates’  Perspective: 
NSHS  San  Diego  and  TNCC 


•  Graduates  are  asked 

-  Have  you  found  it  difficult  to 
understand  and  adjust  to  the  standards 
of  military  behavior,  bearing,  and 
discipline  that  were  expected  at  your 
new  command  (MTF  laboratory) ? 

•  Predicted  response  rates  for  pooled  population  showed 
no  statistically  significant  differences  between  the 
programs 

-  Response  rate  is  percentage  of  graduates  who  found  it 
somewhat  to  very  easy  to  understand  and  adjust 


Graduates  were  asked  to  rank  how  difficult  it  had  been  for  them  to 
understand  and  adjust  to  the  standards  of  military  behavior,  bearing,  and 
discipline  that  were  expected  of  them  at  their  new  command.  Responses  were 
based  on  a  5-point  scale  (l=very  difficult,  2=somewhat  difficult,  3=neither 
difficult  nor  easy,  4=somewhat  easy,  and  5=very  easy).  We  collapsed  these 
responses  into  a  dichotomous  variable:  1  indicated  a  response  of  somewhat  to 
very  easy,  and  0  indicated  a  response  of  neither  difficult  nor  easy  to  very 
difficult. 

Controlling  for  graduate  characteristics,  we  found  no  statistically 
significant  differences  in  graduates’  ease  of  adjustment  associated  with  TNCC 
versus  NSHS  San  Diego  (looking  at  responses  of  somewhat  to  very  easy).  This 
result  was  the  same  when  we  looked  at  interim  findings  for  the  12-month  post¬ 
graduation  surveys. 
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Summary  of  Findings 


•  TNCC  program  associated  with 

-  Higher  ASCP  pass  rates  (quality) 

-  Lower  per-graduate  costs  (cost  efficiency) 

•  TNCC  is  at  least  as  successful  as  in-house 
program  in  terms  of 

-  Attrition  rates  (production  success) 

-  End-user  satisfaction 

-  Student  satisfaction  with  MLT  program 

-  Student  satisfaction  with  many  aspects  of  QOL 

-  Military  bearing 

•  As  of  4  -month  post-graduation  survey,  no  difference,  but 
continue  to  monitor  supervisor  satisfaction  with  12-month 
post-graduation  survey 


We  have  looked  at  the  TNCC  pilot  program  using  several  different 
outcome  measures.  In  summary,  we  found  that  the  TNCC  program  produces 
high-quality  graduates  at  a  lower  per-graduate  cost  when  compared  with  the 
in-house  MLT  training  program. 

The  TNCC  program  accomplishes  this  while  continuing  to  be  at  least 
as  successful  as  the  in-house  program  with  regard  to  attrition  rates,  satisfaction 
of  MTF  lab  supervisors  working  with  the  program  graduates,  student 
satisfaction  with  the  program  and  most  aspects  of  QOL,  incidence  of 
disciplinary  action  among  students,  and  graduates’  military  bearing.1 


1 .  For  military  bearing,  4-month  post-graduation  graduate  and  supervisor  surveys  showed  no 
statistically  significant  difference  with  regard  to  military  bearing.  Interim  results  from  the  12- 
month  post  graduation  supervisor  survey  do  indicate  statistically  lower  satisfaction  with  TNCC 
graduates’  attitude  toward  the  job.  We  will  continue  to  monitor  this  as  we  complete  the  12- 
month  post -graduation  surveys  for  the  remainder  of  MLT  courses  included  in  the  pilot 
evaluation  (to  date,  12-month  post-graduation  surveys  have  been  completed  for  about  half  of 
the  population). 
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Conclusion 


•  How  do  we  weight  the  “success”  criteria? 

-  Cost  savings  subject  to  quality  of  graduate 

-  Other  criteria  signal  need  for  improvement 

•  TNCC  pilot  should  be  considered  successful 

•  Recommendations  for  follow-up 

-  Address  QOL  aspects  for  TNCC  students 

-  Complete  12- month  post-graduation  surveys 

-  Evaluate  long-term  cost  efficiency 

-  Reexamine  set-back  policy  at  NSHS  San  Diego 

-  Collaborate  with  TNCC  to  incorporate  successes  into 
in-house  training  program 


To  determine  the  success  or  failure  of  the  TNCC  pilot  program,  Navy 
Medicine  had  to  establish  the  importance,  or  weight,  that  should  be  given  to  each 
of  the  outcome  measures  being  monitored.  It  was  determined  that  cost  savings 
would  be  considered  the  most  important  criterion,  subject  to  at  least  maintaining 
the  same  graduate  quality  as  NSHS  San  Diego.  The  other  criteria — militarization, 
student  satisfaction  with  the  program,  QOL,  and  end-user  satisfaction — should  be 
used  to  signal  the  need  for  program  changes  and/or  enhancements  rather  than  to 
determine  the  ultimate  success  or  failure  of  the  pilot. 

Based  on  our  evaluation,  the  TNCC  pilot  clearly  achieved  cost-efficiency 
while  maintaining  quality.  Therefore,  we  recommend  that  the  pilot  be  considered  a 
success.  In  addition,  we  recommend: 

•  Addressing  those  aspects  of  QOL  for  which  TNCC  student  satisfaction  was 
lower  than  NSHS  Portsmouth’s  satisfaction  (through  additional  monitoring, 
analysis,  and  focus  groups) 

•  Continuing  to  monitor  satisfaction  of  supervisors  and  graduates  until  all  classes 
have  reached  the  12-month  post-graduation  mark 

•  Conducting  long-term  evaluation  to  determine  impact  of  outsourced  training  on 
life-cycle  training  costs  (follow  graduates  at  least  through  their  next  reenlistment 
decision) 

•  Reevaluating  the  policy  and  use  of  the  set-back  concept  for  remediation  at 
NSHS  San  Diego 

•  Lor  areas  in  which  the  pilot  program  was  particularly  successful,  identifying 
lessons  learned  that  could  be  adopted  by  the  in-house  training  program. 
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Appendix  A:  Logistic  regression  and  estimated 
probabilities 


Many  of  the  outcomes  evaluated  in  this  analysis  are  measured  as  dichoto¬ 
mous  variables,  such  as  the  0/1  measure  of  satisfaction  with  the  MLT  pro¬ 
gram,  or  the  0/1  indicator  for  an  event,  such  as  academic  attrition  or 
disciplinary  action.  To  estimate  the  probability  of  each  outcome  conditional 
on  individual  student  quality  and  demographic  characteristics,  we  use  the 
logistic  model.  The  logistic  model  takes  the  form: 


log 


'  P  - 
_1  -P. 


Ot  +  PlYl  +  PlYl  +  ...  +  QkXk 


(1) 


where  P  is  the  probability  of  some  event,  such  as  being  satisfied  or  attriting 
from  the  program.  X  represents  the  k  independent  variables — individual 
student  quality  and  demographic  characteristics;  a  and  P  represent  coeffi¬ 
cients  to  be  estimated.  Using  the  estimated  coefficients,  we  can  calculate 
the  predicted  probability  of  an  event  as: 

P  =  - - -  ■  (2) 

(1  +  exp{  Ot  +  P  lXl  +  P 2X2  +  ...  +  P  kXk)) 

For  each  outcome  measure,  the  vector  of  independent  variables  includes: 

•  Treatment  group  indicator  (indicates  whether  the  student  attended 
Thomas  Nelson  Community  College  (TNCC)  or  the  control  site) 

•  FY  1999  indicator  (indicates  whether  the  student  attended  C-school 
during  the  first  year  of  the  pilot — the  startup  phase) 

•  Selected  demographic  and  student  quality  variables  as  appropriate 
for  the  specific  outcome  and  population  of  interest. 

By  conditioning  our  estimates  on  individual  characteristics,  we  control  for 
the  effect  that  any  differences  in  these  characteristics  might  have  on  the  out¬ 
come.  The  coefficient  on  the  treatment  variable  represents  the  estimated 
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effect  of  the  pilot  program  (or  treatment),  holding  all  other  characteristics 
constant.  Combined  with  the  significance  test  on  the  estimated  coefficient, 
we  determine  whether  the  TNCC  program  is  associated  with  a  significant 
positive  or  negative  effect  on  the  outcome  measure,  relative  to  the  control 
group. 

For  each  outcome,  we  estimate  the  logistic  regression  model  (equation  1) 
for  the  pooled  population  (students  or  graduates  from  both  TNCC  and  the 
appropriate  control  group,  either  NSF1S  SD  or  NSFIS  Portsmouth).  Using 
the  estimated  coefficients,  we  calculate  the  predicted  probability  for  each 
student  in  the  pooled  population  twice:  once  turning  the  treatment  variable 
on  (=1)  for  the  entire  population,  and  the  second  time  turning  the  treatment 
variable  off  (=0)  for  the  entire  population.1  The  means  of  the  resulting  two 
sets  of  probabilities  reflect  the  average  predicted  probability  of  sending  the 
pooled  population  to  TNCC  versus  sending  that  same  population  (with 
exactly  the  same  characteristics)  to  the  control  site,  controlling  for  all  the 
characteristics  included  as  independent  variables  (X’s). 


1.  To  calculate  the  predicted  probabilities,  we  use  equation  2  with  the  actual  X 
values  for  each  student  as  well  as  the  estimated  a  and  P’s. 
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Appendix  B:  Detailed  accounting  of  courses 
included  in  pilot  evaluation 

The  pilot  phase  was  determined  by  the  time  line  required  to  complete  four 
TNCC  MLT  courses.  The  first  course  started  in  January  1999.  The  fourth 
and  final  TNCC  MLT  course  included  in  the  pilot  had  a  completion  date  of 
July  2001.  The  complete  set  of  courses  in  the  pilot  evaluation  fall  into  three 
groups: 

•  The  treatment  group — TNCC  MLT  courses  started  between  January 
1999  and  July  2000 

•  The  MLT  program  control  group — NSJJS  San  Diego  MLT  courses 
started  between  January  1999  and  July  2000 

•  The  quality-of-life  (QOL)  control  group — NSHS  Portsmouth 
selected  C-schools 

—  Pharmacy  Technician  courses  started  between  January  1999  and 
January  2001 

—  X-ray  Technician  courses  started  between  September  1998  and 
May  2000. 

Table  1  lists  the  specific  courses  in  each  group  (the  convening  date,  gradu¬ 
ation  date,  student  load,  and  number  of  graduates).1 


1 .  For  NSHS  Portsmouth  X-ray  and  Pharmacy  Technicians,  we  do  not  have  spe¬ 
cific  data  on  student  load  (course  enrollees)  and  graduates.  For  NSHS  Ports¬ 
mouth  graduates,  we  report  the  number  of  end-of-course  graduate  surveys 
received.  Data  reported  for  NSHS  SD  and  TNCC  MLT  courses  were  provided 
by  the  respective  MLT  programs. 
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Table  1.  Eligible  courses  for  TNCC  MLT  pilot  evaluation 


Class 

Convening 

date 

Graduation 

date 

Student 

load 

Graduates1 

Treatment  group 

MLT  TNCC 

99005 

04-Jan-99 

1  7-Jan-00 

18 

18 

MLT  TNCC 

99010 

01  -J  u  1-99 

1 3 -J  u  1-00 

47 

42 

MLT  TNCC 

00005 

04-Jan-00 

1  6-Jan-01 

44 

39 

MLT  TNCC 

00010 

03-Jul-00 

1  6-J  u  1-01 

41 

37 

Total 

150 

136 

MLT  in 

■house  program  control  group  (NSHS  San  Diego) 

MLT  SD 

99010 

1  7-Feb-99 

1  7-Mar-00 

35 

23 

MLT  SD 

99015 

27-May-99 

28-Jun-OO 

23 

14 

MLT  SD 

99020 

02-Sep-99 

04-0ct-00 

40 

21 

MLT  SD 

00005 

09-Dec-99 

24-Jan-01 

41 

25 

MLT  SD 

00010 

29-Mar-00 

27-Apr-01 

36 

21 

MLT  SD 

00015 

05-Jul-00 

03-Aug-01 

27 

18 

Total 

202 

122 

QOL  control  group  (NSHS  Portsmouth) 

Pharm 

P-135 

05-Jan-99 

1 3-Jun-99 

n.a. 

31 

Pharm 

P-136 

05-Apr-99 

1  5-Sep-99 

n.a. 

n.a. 

Pharm 

P-137 

23-Jun-99 

1  7-Dec-99 

n.a. 

31 

Pharm 

P-138 

22-Sep-99 

23-Mar-00 

n.a. 

36 

Pharm 

P-139 

03-Jan-00 

21  -Jun-00 

n.a. 

31 

Pharm 

P-140 

03-Apr-00 

1 3-Sep-00 

n.a. 

26 

Pharm 

P-141 

26-Jun-00 

14-Dec-00 

n.a. 

17 

Pharm 

P-142 

22-Sep-00 

23-Mar-01 

n.a. 

20 

Pharm 

P-143 

03-Jan-01 

21  -Jun-01 

n.a. 

18 

Adv  X-ray 

53 

28-Sep-98 

21 -Oct-99 

n.a. 

14 

Adv  X-ray 

54 

1  9-Apr-99 

21 -May-00 

n.a. 

18 

Adv  X-ray 

55 

25-Oct-99 

1  6-Nov-00 

n.a. 

6 

Adv  X-ray 

56 

1 5-May-00 

07-Jun-01 

n.a. 

13 

n.a.  261 


1.  We  did  not  have  data  on  NSHS  Portsmouth  Pharmacy  Tech  and  X-ray  Tech  graduates.  Numbers 
reported  are  graduate  respondents  to  the  Pharmacy  and  X-ray  Technician  end-of-course  surveys. 
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Appendix  C:  Population  statistics  and  logistic 
regression  results  for  ASCP  comparison 

For  quality  of  product,  we  compare  the  American  Society  of  Clinical 
Pathologists  (ASCP)  registry  exam  pass  rates  of  TNCC  MLT  graduates  and 
NSF1S  San  Diego  MLT  graduates.1 2  Tables  1  and  2  provide  population 
means  and  logistic  regression  results.*" 

Table  1.  Population  means:  TNCC  and  NSHS  SD  graduates  that  sat  for  the  ASCP 


Variable 

NSHS  SD 
(n=121) 

TNCC 

(n=135) 

Pooled 

(n=256) 

Passed  ASCP  exam3 

0.33 

0.71 

0.53 

Student  quality: 

-  Was  set  back3 

0.21 

0.00 

0.10 

-  AFQT  score 

65.22 

65.33 

65.28 

-  A-school  GPA 

88.18 

88.56 

88.38 

-  Has  basic  MLT  NEC3 

0.07 

0.03 

0.05 

-  Missing  some  prerequisite3 

0.07 

0.03 

0.05 

-  Had  previous  college 

0.58 

0.65 

0.62 

Demographic: 

-  Gender  (female)3 

0.24 

0.39 

0.32 

-  Race  (not  Caucasian) 

0.69 

0.66 

0.67 

-  Age  i  n  years 

23.77 

24.50 

24.16 

-  Married 

0.39 

0.45 

0.42 

-  Has  children 

0.33 

0.39 

0.36 

a.  Difference  in  population  means  is  statistically  significant. 


1 .  All  MLT  graduates  from  TNCC  and  NSHS  San  Diego  were  expected  to  sit  for 
the  ASCP  exam  as  part  of  the  pilot  evaluation.  One  graduate  from  each  pro¬ 
gram  did  not  sit  for  the  exam.  Therefore,  these  observations  were  excluded 
from  the  population  of  interest  for  product  quality. 

2.  Force  fill  and  paygrade  were  not  included  in  the  ASCP  model.  There  was  sig¬ 

nificant  correlation  between  force  fill,  paygrade,  and  age.  Controlling  for  all 
three  or  any  combination  of  two  did  not  add  any  explanatory  value  to  the 
model.  Age  was  the  only  variable  that  was  individually  significant. 
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Table  2.  Logistic  regression  results:  pass  ASCP  exam  (=1) 

Variable  Coefficient  Standard  Error 


Constant 

-14.04993 

3.5672 

Treatment  group  (TNCC=1) 

1.6485a 

0.3674 

FY  1 999  (startup) 

0.6679c 

0.3687 

Student  quality: 

-  Was  set  back 

-0.3836 

0.6721 

-  AFQT  score 

0.0571a 

0.0132 

-  A-school  GPA 

0.0796b 

0.0416 

-  Has  basic  MLT  NEC 

0.6843 

0.7842 

-  Missing  some  prerequisite 

-1 .0046 

0.8352 

-  Had  previous  college 

-0.7283c 

0.3828 

Demographic: 

-  Gender  (female) 

1 .551 9a 

0.4091 

-  Race  (not  Caucasian) 

0.5907 

0.3756 

-  Age  i  n  years 

0.0704c 

0.0431 

-  Married 

0.6090 

0.4290 

-  Has  children 

-0.1768 

0.4365 

Mean  predicted  pass  rated: 

-  TNCC 

0.66 

-  NSHS  San  Diego 

0.37 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 

d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Appendix  D:  Population  statistics  and  logistic 
regression  results  for  academic  attrition 
comparison 

For  production  success,  we  compare  the  rates  of  academic  attrition  across 
the  two  MLT  programs  (TNCC  and  NSFIS  San  Diego).  Because  the  set¬ 
back  concept  was  available  only  at  NSFIS  San  Diego,  we  calculate  attrition 
using  two  different  models:1 

•  Academic  attrition  among  all  enrollees,  where  academic  attrition 
includes  set  backs  (SBs)  and  disenrollments  (DEs).  In  determining 
the  population  of  “all  enrollees,”  we  count  students  who  were  SB  as 
two  separate  enrollments  (once  in  original  class,  outcome  SB,  and 
once  in  the  subsequent  class,  outcome  graduate  or  fail  course). 

•  Academic  attrition  among  unique  enrollees  (or  unique  student  popu¬ 
lation).  Here  academic  attrition  includes  only  DEs.  In  determining 
the  population  of  “unique  enrollees,”  we  count  students  who  were 
SB  as  only  one  enrollment  (or  one  unique  student).  To  do  this,  we 
exclude  observations  for  which  the  class  outcome  was  an  SB. 

The  following  tables  provide  population  means  and  logistic  regression 
results  for  these  two  attrition  models.  Tables  1  and  2  refer  to  academic  attri¬ 
tion  for  all  enrollees  (includes  set  backs).  Tables  3  and  4  refer  to  academic 
attrition  for  unique  enrollees  (excluding  set  backs). 


1.  In  both  models,  we  exclude  observations  (students  or  enrollees)  that  resulted 
in  nonacademic  (medical,  attitude,  legal,  etc.)  attrition.  We  assume  that  non- 
academic  attrition  is  purely  random. 
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Table  1.  Population  means  for  TNCC  and  NSHS  SD  enrollees  (excluding  nonacademic  attrition) 

NSHS  SD  TNCC  Pooled 

Variable _ (n=186)  (n=146)  (n=332) 


Academic  Attrition  (includes  SB  and  DE)a 

0.34 

0.07 

0.22 

Student  quality: 

-  Was  set  back3 

0.23 

0.00 

0.13 

-  AFQT  score 

63.13 

64.92 

63.92 

-  A-school  GPAa 

86.50 

87.81 

87.08 

-  Has  basic  MLT  NEC 

0.05 

0.03 

0.04 

-  Missing  some  prerequisite3 

0.10 

0.03 

0.07 

-  Had  previous  college3 

0.53 

0.63 

0.58 

Demographics: 

-  Paygrade  of  E-3  to  E-63 

0.70 

0.90 

0.79 

-  Gender  (female)3 

0.20 

0.36 

0.27 

-  Race  (not  Caucasian) 

0.69 

0.65 

0.67 

-  Married 

0.37 

0.43 

0.40 

-  Has  children 

0.31 

0.37 

0.34 

Quality  of  peers: 

-  %  of  classmates  having  basic  MLT  NEC3 

0.05 

0.03 

0.04 

-  %  of  classmates  with  previous  college3 

0.52 

0.62 

0.57 

a.  Difference  in  population  means  is  statistically  significant. 
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Table  2.  Logistic  regression  results:  academic  attrition  including  set  backs 
Variable  Coefficient  Standard  Error 


Constant 

28.9776a 

4.5008 

Treatment  group  (TNCC=1) 

-2.17003 

0.6742 

FY  1 999  (startup) 

0.3594 

0.5743 

Student  quality: 

-  Was  set  back 

-0.5633 

0.4640 

-  AFQT  score 

0.0069 

0.0140 

-  A-school  GPA 

-0.3343a 

0.0515 

-  Has  basic  MLT  NEC 

-1.4985 

1.2452 

-  Missing  some  prerequisite 

1.4327b 

0.5964 

-  Had  previous  college 

-0.3789 

0.3998 

Demographics: 


-  Paygrade  of  E-3  to  E-6 

-0.7250c 

0.4060 

-  Gender  (female) 

-0.9669° 

0.5576 

-  Race  (not  Caucasian) 

-0.4662 

0.4490 

-  Married 

-0.5298 

0.4709 

-  Has  children 

-0.1369 

0.4965 

Quality  of  peers: 

-  %  of  classmates  having  basic  MLT  NEC  5.9846  5.5566 

-  %  of  classmates  with  previous  college  -0.9694  2.2259 


Mean  predicted  academic  attrition  rated: 

-  TNCC  0.10 

-  NSHS  San  Diego  0.29 


a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 

d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Table  3.  Population  means  for  TNCC  and  NSHS  SD  unique  enrolleesa 
(excluding  nonacademic  attrition) 


NSHS  SD  TNCC  Pooled 
Variable  (n=144)  (n=146)  (n=290) 


Academic  attrition  (DEs  only)b 

0.15 

0.07 

0.11 

Student  quality: 

-  Was  set  backb 

0.30 

0.00 

0.15 

-  AFQT  score 

64.38 

64.92 

64.65 

-  A-school  GPA 

87.21 

87.81 

87.51 

-  Missing  some  prerequisite1’ 

0.08 

0.03 

0.06 

-  Had  previous  college 

0.55 

0.63 

0.59 

Demographics: 

-  Paygrade  of  E-3  to  E-6b 

0.73 

0.90 

0.81 

-  Gender  (femaler 

0.22 

0.36 

0.29 

-  Race  (not  Caucasian) 

0.68 

0.65 

0.67 

-  Married 

0.39 

0.43 

0.41 

-  Has  children 

0.32 

0.37 

0.35 

Quality  of  peers: 

-  %  of  classmates  having  basic  MLT  NECb 

0.05 

0.03 

0.04 

-  %  of  classmates  with  previous  college13 

0.51 

0.62 

0.57 

a.  The  population  of  unique  enrollees  counts  set  backs 

only  once  (thus  enrollments 

that  end  in  a  set  back  are  excluded  from  the  sample), 
b.  Difference  in  population  means  is  statistically  significant. 
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Table  4.  Logistic  regression  results:  academic  attrition  excluding  set  backs 


Variable 

Coefficient 

Standard  error 

Constant 

41.88123 

8.4989 

Treatment  group  (TNCC=1) 

1.4193 

1.1862 

FY  1999  (startup) 

-0.4246 

1.0240 

Student  quality: 

-  Was  set  back 

2.6901a 

0.8376 

-  AFQT  score 

0.0123 

0.0240 

-  A-school  GPA 

-0.4853a 

0.0987 

-  Missing  some  prerequisite 

0.7371 

0.9107 

-  Had  previous  college 

-0.2226 

0.6810 

Demographics: 

-  Paygrade  of  E-3  to  E-6 

-1 .31 59b 

0.6304 

-  Gender  (female) 

-1.2435 

1.0249 

-  Race  (not  Caucasian) 

-1.3572c 

0.7872 

-  Married 

-0.5431 

0.7433 

-  Has  children 

-0.2315 

0.7649 

Quality  of  peers: 

-  %  of  classmates  having  basic  MLT  NEC 

9.7896 

9.3801 

-  %  of  classmates  with  previous  college 

-6.3501 c 

3.7915 

Mean  predicted  academic  attrition  rated: 

-  TNCC 

-  NSHS  San  Diego 

0.10 

0.11 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 

cl.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  “treatment"  coefficient  is  significant. 
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Appendix  E:  Cost  comparison  methodology  and 
supporting  analysis 


The  following  tables  provide  details  of  the  costing  methodology  for  MLT 
programs  at  NSHS  San  Diego  and  TNCC  (overhead  provided  by  NSHS 
Portsmouth). 

Tables  1  and  2  break  down  the  costs  associated  with  MLT  training  at  NSHS 
SD  (table  1)  and  TNCC  (table  2).  In  addition,  table  2  provides  a  comparison 
between  the  two  programs  (percentage  difference).  Costs  in  these  tables  are 
divided  into: 

•  Activity  specific — those  overhead  costs  that  are  spread  across  all 
courses  taught  at  or  through  the  specific  NSHS  (academic,  adminis¬ 
trative,  and  base  operating  support,  and  G&A).  All  activity-specific 
cost  data  (and  activity  specific  student  load  data)  were  provided  by 
MED-51. 

•  Course  specific — those  costs  associated  directly  with  the  teaching 
and  administration  of  the  MLT  courses.  These  include  active  duty 
Navy  MLT  instructor  and/or  course  oversight  and  any  associated 
TAD/training  dollars  for  these  people,  MLT  student  expenses  (lab 
agents,  books,  equipment,  reproduction,  etc.),  tuition,  and  transpor¬ 
tion.1 


1.  AD  Navy  instructor/oversight  data  (numbers  and  composite  pay)  were  pro¬ 
vided  by  MED-5 1 .  TAD  and  training  budget  for  TNCC  AD  Navy  instructors/ 
oversight  was  provided  by  TNCC  contract  officer  (none  reported  for  NSHS 
SD).  MLT  student  expenses  for  NSHS  SD  and  TNCC  were  provided  by 
MED-51  and  TNCC  contract  officer,  respectively.  NSHS  SD  tuition  (paid  to 
George  Washington  University)  was  reported  by  NSHS  Bethesda  (responsible 
party  for  GW  contract).  Tuition  for  TNCC  includes  total  invoice  cost  (tuition, 
books,  office  space,  graduation  ceremony,  etc.)  and  contract  incentive  fee  paid 
to  TNCC  (data  provided  by  TNCC  contract  officer).  Transportation  cost  is 
pertinent  to  TNCC  program  only  and  was  provided  by  TNCC  contract  officer. 
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Additional  data  in  the  table  include: 

•  Each  Activity’s  (NSHS)  average  daily  student  load  (ADS) — used  to 
determine  the  per-student  academic  and  administrative  support  costs. 

•  Course  length  in  days — used  to  determine  the  G&A  and  BOS  support 
cost  per  student  (because  both  are  reported  in  per  student  daily  rate). 

•  Three  course-specific  student  measures 

—  Unique  enrollees,  a  number  that  reflects  the  unique  student  load  for 
NSHS  SD  by  counting  those  students  who  were  set  back  (enrolled  in 
an  initial  class  and  set  back  to  be  enrolled  in  a  subsequent  class)  only 
once.  This  measure  is  required  to  determine  the  total  tuition  cost  for 
NSHS  SD  because  tuition  is  paid  only  once  for  each  student,  regard¬ 
less  of  whether  he/she  is  set  back. 

—  Estimated  graduates,  based  on  our  analysis  of  academic  attrition 
rates.  We  start  with  each  program’s  actual  enrollment,  delete  those 
enrollees  who  were  attrited  for  nonacademic  reasons  (medical, 
administrative,  attitude,  etc.),  and  then  apply  the  predicted  academic 
attrition  rate  for  each  program  (predicted  rates  reflect  the  academic 
attrition  rate  we  would  expect  to  see  by  sending  the  pooled  popula¬ 
tion  to  TNCC  vs.  NSHS  SD — controlling  for  enrollee  demographic 
and  student  quality  characteristics).  This  measure  is  used  to  calculate 
the  per-graduate  cost  for  all  course-specific  cost  factors. 

—  Actual  graduates,  provided  for  comparison  only — not  used  for  any 
cost  calculations. 


2.  Course  length  reported  by  MED-5 1  did  not  reflect  actual  course  length  as  reported 
in  NITRAS  (rather  it  reflected  the  NITRAS  planned  course  length).  As  a  result  of 
a  TNCC  error,  the  four  TNCC  pilot  courses  that  we  evaluated  were  actually  on 
average  20  days  shorter  than  the  NSHS  SD  course  (this  represents  approximately 
14  training  days).  Twenty  days  is  equivalent  to  $2,148  in  student  programming 
rate  dollars  (using  average  of  FY 2000  and  FY 200 1  programming  rate  of  $39,200) . 
Additionally,  this  would  reduce  those  NSHS  PTS/TNCC  costs  that  are  calculated 
as  a  daily  rate  (G&A  overhead  per  student  and  base  operating  support  per  student). 
Because  this  was  an  error  and  will  be/has  been  corrected  for  subsequent  TNCC 
courses,  we  did  not  include  these  savings  in  our  analysis. 
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Finally,  for  all  course-specific  data  provided  to  us  by  fiscal  year  or  contract 
year,  we  had  to  determine  what  portion  of  the  FY  or  CY  costs  were  applicable 
to  the  pilot  phase  courses  (those  courses  that  fell  within  the  pilot  evaluation; 
see  appendix  A  for  a  complete  listing).  To  do  this,  we  had  to  determine  the 
amount  of  overlap  between  pilot  evaluation  courses  and  non-pilot-evaluation 
courses.  These  data  are  provided  in  table  3.  We  determine  the  amount  of  over¬ 
lap  in  terms  of  enrollees  (percentage  of  total  enrollees  in  FY  or  CY  that  are 
represented  by  pilot  evaluation  vs.  non-pilot-evaluation  courses). 
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Table  1:  NSHS  SD  MLT  training  cost  based  on  estimated  graduation  rates  (controlling  for  enrollee  characteristics)3 


NSHS  San  Diego 


1999_ 2000_ 2001  Ave.  or  tot.  Per  grad. 


Activity  Specific: 

Average 

Activity  Academic  Support1’ 

$1,424,355 

$1,066,025 

$1,632,728 

$1,374,369 

$3,175 

Activity  Administrative  Support13 

$2,316,226 

$2,222,350 

$1,838,086 

$2,125,554 

$4,910 

Annual  Activity  ADS  Loadc 

417.00 

475.30 

406.50 

432.93 

Subtotal: 

Average 

$8,084 

G&A  overhead  per  Student  (daily)d 

$3.61 

$3.12 

$4.09 

$3.61 

$1,331 

Base  Operating  Support  per  Student  (dai ly)d 

$5.99 

$4.51 

$1.23 

$3.91 

$1,443 

Course  Length  (days)0 

Subtotal: 

369 

369 

369 

369.00 

$2,774 

Activity  Subtotal: 

$10,858 

Course  Specific: 

(0.38*FY99) 

(0.97*FY00) 

(0.5*FY01 ) 

Total 

AD  Navy  Instructors*3 

$264,283 

$511,069 

$265,023 

$1,040,374 

$7,849 

Classroom  Expenses  (SMART)* 

$79,657 

$237,335 

$130,594 

$447,585 

$3,377 

Tuition  (GW  for  SD  /  TNCC  for  PTS)8 

- 

- 

- 

$119,250 

$900 

Transportation  (TNCC  only) 

$0 

$0 

$0 

$0 

$0 

TAD  and  Training  (TNCC  only) 

$0 

$0 

$0 

$0 

$0 

Unique  Enrollees  (count  set  backs  once) 

159 

Estimated  Graduates  (for  entire  pilot) 

133 

Actual  Graduates  (for  comparison  only) 

122 

Course  Subtotal: 

$12,126 

Total  Activity  and  Course  Specific: 

Total 

$22,984 

Table  Notes: 

a)  Does  not  include  student  salary  and  ,  therefore,  does  not  capture  cost  associated  with  set  backs. 

b)  Source  MED-51 :  AD  Navy  staffing  data  from  NSHS  Activity  Manning  Documents  (AMDs).  Staffing  is  multiplied  by  Navy  composite  pay  (specific  to  each 
grade  and  FY)  to  calculate  total  cost  of  AD  staff. 

c)  Source  MED-51  (AOB  -  NITRAS). 

d)  Source  MED-51  (SMART).  These  rates  were  provided  as  daily,  and  in  some  cases  weekly,  per  student  rates.  We  report  them  in  daily,  per  student,  rates.  These 
rates  are  multiplied  by  course  length  (in  days)  to  compute  the  per-student  cost. 

e)  Source  MED-51  (DAYS  -  NITRAS). 

f)  Source  MED-51  (SMART):  Includes  supply  cost  (all  EOE  not  "W,  Q,  U,  A,  1"),  equipment  cost  (EOE  "W"),  purchased  services  (EOE  "Q"),  and  civilian  pay 
(EOE,  "U").  Based  on  discussions  with  NSHS  San  Diego  comptroller,  it  was  determined  that  equipment  costs  (capital  investments)  should  be  averaged  over  a  5- 
year  time  horizon.  MED-51  provided  equipment  costs  for  FY1 997-FY2001  (the  average  being  $1 05,  772).  This  5-year  average  was  substituted  for  eachFY-spe- 
cific  equipment  cost  when  calculating  NSHS  SD's  classroom  expenses  for  FY1999-FY2001 . 

g)  Tuition  for  NSHS  SD  is  paid  to  Georg  Washinton  University  at  a  rate  of  $750  per  enrollee  (set  backs  are  not  charged  tuition  twice).  Total  tuition  is  equal  to 
$750  x  "unique  enrollees."  The  total  tuition  is  then  divided  by  "estimated  graduates"  to  get  a  per-graduate  rate. 
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Table  2:  NSHS  PTS/TNCC  MLT  training  cost  based  on  estimated  graduation  rates  (controlling  for  enrollee  characteristics)3 

_ NSHS  PTS/TNCC _ 

Percentage 


1999 

2000 

2001 

Ave.  or  tot. 

Per  grad. 

Difference11 

Activity  Specific: 

Average 

Activity  Academic  Support6 

$386,748 

$427,225 

$324,421 

$379,465 

$1,688 

Activity  Administrative  Supportc 

$1,133,664 

$1,502,025 

$1,906,864 

$1,514,184 

$6,736 

Annual  Activity  ADS  Loadd 

200.00 

245.50 

228.90 

224.80 

Subtotal : 

Average 

$8,424 

-4% 

C&A  overhead  per  Student  (daily)6 

$4.00 

$2.92 

$3.57 

$3.50 

$1,290 

Base  Operating  Support  per  Student  (daily)6 

$4.14 

$5.24 

$4.01 

$4.46 

$1,647 

Course  Length  (days/ 

369 

369 

369 

369.00 

Subtotal : 

$2,937 

-6% 

Activity  Subtotal: 

$11,361 

-5% 

Course  Specific: 

(1  *FY99) 

(1  *FY00) 

(0.57  *  FY01) 

Total 

AD  Navy  Instructors6 

$204,965.00 

$213,375.00 

$120,831.45 

$539,171.45 

$4,086 

Classroom  Expensesf 

$5,000.00 

$5,000.00 

$1,691.76 

$11,691.76 

$89 

Tuition  (GW  for  SD  /  TNCC  for  PTS)8 

- 

- 

- 

$783,006.43 

$5,934 

Transportation  (TNCC  only) 

$4,056.00 

$4,166.40 

$3,107.64 

$11,330.04 

$86 

TAD  and  Training  (TNCC  only)' 

$4,600.00 

$4,600.00 

$2,635.68 

$11,835.68 

$90 

Unique  Enrollees  (count  set  backs  once) 

150 

Estimated  Graduates  (for  entire  pilot) 

132 

Actual  Graduates  (for  comparison  only) 

136 

Course  Subtotal: 

$10,285 

15% 

Total  Activity  and  Course  Specific: 

Total 

$21,646 

6% 

Table  Notes: 

a)  Does  not  include  student  salary  and  ,  therefore,  does  not  capture  difference  in  cost  associated  with  set  backs  at  NSHS  SD. 

b)  Percent  difference  is  equal  to  (SD-PTS)/SD).  Therefore,  positive  value  indicates  the  percentage  by  which  the  NSHS  PTS/TNCC  program  costs  are  less 
than  SD. 

c)  Source  MED-51 :  AD  Navy  staffing  data  from  NSHS  Activity  Manning  Documents  (AMDs).  Staffing  is  multiplied  by  Navy  composite  pay  (specific  to 
each  grade  and  FY)  to  calculate  total  cost  of  AD  staff. 

d)  Source  MED-51  (NITRAS). 

e)  Source  MED-51  (SMART).  These  rates  were  provided  as  daily,  and  in  some  cases  weekly,  per  student  rates.  We  report  them  in  daily,  per  student,  rates. 
These  rates  are  multiplied  by  course  length  (in  days)  to  compute  the  per-student  cost. 

f)  Source  TNCC  program  director/contract  officer:  Reported  as  supplies  and  equipment  for  Navy-specific  training. 

g)  Source  TNCC  program  director/contract  officer:  Tuition  for  TNCC  includes  all  invoice  costs  and  incentive  award  fees. Total  tuition  is  reported  for  the  4 
TNCC  MLT  pilot  courses.  The  total  tuition  is  then  divided  by  "estimated  graduates"  to  get  a  per-graduate  rate. 

h)  Source  TNCC  program  director/contract  officer,  via  Comptroller  Head,  Financial  Management  Code  1 1 . 

i)  Source  TNCC  program  director/contract  officer.  Reflects  Navy  budget  for  TNCC  AD  instructors  TAD  and  Training  (not  captured  in  SMART). 
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Table  3:  Non-pilot  courses  overlap  with  pilot  courses  (by  percentage  of  enrollees  in  FY)* 


Start 

Class  months  in 

FY 

Total 

Enrollee  in  FY 

End 

1999 

2000 

2001 

enrollees 

1999 

2000 

2001 

NSHS  San  Diego  MLT 

Apr-98 

Apr-99 

7.0 

23 

12 

J  u  1-98 

Ju  1-99 

10.0 

23 

18 

Nov-98 

Nov-99 

11.0 

2.0 

23 

19 

4 

Feb-99 

Feb-00 

7.5 

5.5 

35 

20 

15 

Jun-99 

Jun-00 

4.0 

9.0 

23 

7 

16 

Sep-99 

Sep-00 

1.0 

12.0 

40 

3 

37 

Dec-99 

Jan-00 

10.0 

4.0 

41 

29 

12 

Apr-00 

01 -Apr 

6.0 

7.0 

36 

17 

19 

J  u  1-00 

01-Jul 

3.0 

10.0 

27 

6 

21 

Oct-OO 

01  -Nov 

11.0 

18 

15 

01  -Feb 

01  -Feb 

8.0 

26 

16 

01  -May 

01  -May 

5.0 

38 

15 

01  -Aug 

01  -Aug 

2.0 

37 

6 

Total  enrollees 

in  FY: 

80 

123 

103 

In  pilot  time  frame: 

30 

120 

52 

Out  of  pilot  time  frame: 

50 

4 

52 

Percent  of  total 

in  pilot: 

38% 

97% 

50% 

TNCC  MLT 

Jan-99 

Jan  00  9.0 

4.0 

18 

12 

6 

J  u  1-99 

J  u  1-00  3.0 

10.0 

47 

11 

36 

Jan-02 

9.0 

3.5 

44 

32 

12 

Jul-02 

3.0 

9.5 

41 

10 

31 

Jan-01 

Jan-02 

9.0 

33 

23 

J  u  1-01 

Jul-02 

3.0 

41 

9 

Total  enrollees 

in  FY: 

23.3 

83.2 

75.8 

In  pilot  time  frame: 

23.3 

83.2 

43.5 

Out  of  pilot  time  frame: 

0.0 

0.0 

32.3 

Percent  of  total 

in  pilot: 

1 00% 

1 00% 

57% 

*  Grey  shading  indicates  courses  that  are  not  included 

in  the  pilot  evaluation  (but  may  overlap  with  the  pilot  time  frame— January  1 999  to  August 

2001). 
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Appendix  F :  Population  statistics  and  logistic 
regression  results  for  student  satisfaction  with 
program  (end-of-course  survey) 

For  satisfaction  with  the  MLT  program,  we  compare  rates  of  satisfaction  with 
several  aspects  of  the  MLT  program  across  TNCC  MLT  graduates  and  NSF1S 
San  Diego  MLT  graduates.  Satisfaction  was  measured  using  an  end-of-course 
(EOC)  survey  instrument.  Table  1  lists  those  questions  that  dealt  specifically 
with  student’s  satisfaction  with  the  MLT  program. 

Table  1.  End-of-course  survey  questions:  satisfaction  with  MLT  program 

_ Short  name  Survey  question,  "How  satisfied  are  you  with..." _ 

Quality  of  instruction:  The  quality  of  the  instruction  in  the  classroom 
Instuctors  prep:  The  instructor's  preparation  and  organization 
Instuctors  ability:  The  instructor's  ability  to  explain  how  the  course's  subject  matter  applies  to  the  job  of  an  MLT 
Monitoring  of  inst:  The  monitoring  and  supervision  of  the  instructors  by  the  supervising  staff 
Access  to  inst:  Access  to  instructors 
Access  to  computer:  Access  to  the  Computer  Lab  at  NSHS 

Quality  of  equip:  The  quality  of  the  educational  equipment,  including  visual  aids  and  laboratory  equipment 
Quality  of  texts:  The  quality  of  the  textbooks 
Clinical  rotations:  The  quality  of  instruction  during  clinical  rotations 
Prep  for  clinical:  Program's  adequacy  in  preparing  you  for  clinical  rotations 
Prep  to  be  MLT:  The  program's  assistance  in  preparing  you  to  be  an  entry  level  lab  technician 
Prep  for  ASCP  exam:  The  program's  assistance  in  preparing  you  to  take  the  Registry  Exam 
Prep  for  A. A.  degree:  The  program's  assistance  in  preparing  you  to  receive  an  Associate  degree 
Fellow  students:  The  belief  that  your  fellow  students  wanted  to  be  in  this  particular  program 
Discipline:  The  good  order  and  discipline  of  the  classroom  environment 
Overall:  The  MLT  program  overall 
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Valid  responses  to  the  survey  questions  were  measured  on  a  5-point  Likert 
scale  (l=very  satisfied,  2=somewhat  satisfied,  3=neither  satisfied  or  dissat¬ 
isfied,  4=somewhat  dissatisfied,  5=very  dissatisfied).  For  our  analysis,  sat¬ 
isfaction  responses  were  collapsed  into  a  dichotomous  variable  where 
satisfaction  was  determined  to  be  a  response  of  somewhat  to  very  satisfied 
(1  or  2)  versus  not  satisfied  (a  response  of  3,  4,  or  5). 

Tables  2  and  3  provide  population  means  and  logistic  regression  results  for 
each  aspect  of  program  satisfaction. 


Table  2.  Population  means:  TNCC  and  NSHS  SD  graduates  with  completed  EOC  survey 


Variable 

NSHS  SD 
(n=109) 

TNCC 
(n=1 25) 

Pooled 

(n=234) 

Very  to  somewhat  satisfied  with  orders  upon 

graduation 

0.72 

0.78 

0.75 

Student  quality: 

-  A-school  CPA 

88.07 

88.40 

88.24 

-  A-school  CPA  is  above  average  for 
one's  classmates 

0.50 

0.50 

0.50 

-  Missing  some  prerequisite 

0.06 

0.02 

0.04 

-  Had  previous  college 

0.57 

0.64 

0.61 

Demographics: 

-  Paygrade  of  E-1  -  E-3a 

0.47 

0.29 

0.37 

-  Gender  (female)3 

0.27 

0.40 

0.34 

-  Age  in  years3 

23.50 

24.48 

24.03 

-  Race  (not  Caucasian) 

0.67 

0.66 

0.67 

-  Married 

0.39 

0.45 

0.42 

-  Has  children 

0.53 

0.62 

0.58 

-  Resides  with  family 

0.29 

0.36 

0.33 

-  Lives  on  base 

0.44 

0.41 

0.42 

a.  Difference  in  population  means  is  statistically 

significant. 
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Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ... 


Variable 

Quality  of  instruction 

Instructors'  prep 

Instructors' 

ability 

Coefficient 

S.E. 

Coefficient  S.E. 

Coefficient 

S.E. 

Constant 

1.9256 

4.4044 

4.4488  4.0938 

-4.2530 

4.1342 

Treatment  group  (TNCC=1) 

1.3023b 

0.6137 

1.791 1a  0.5797 

0.91 09c 

0.5599 

FY  1999  (startup) 

1.0748c 

0.5703 

0.5870  0.4798 

-0.2518 

0.4938 

Very  to  somewhat  satisfied 

with  orders  upon  graduation 

0.2975 

0.4862 

0.9681 b  0.4545 

0.4042 

0.4592 

Student  quality: 

-  A-school  GPA 

-0.0167 

0.0490 

-0.0724  0.0463 

0.0387 

0.0461 

-  A-school  GPA  is  above 

average  for  one's  classmates 

0.2959 

0.4706 

1.11 68a  0.4527 

0.3850 

0.4371 

-  Missing  some  prerequisite 

-0.7349 

0.8219 

-0.9140  0.8245 

-0.4439 

0.9075 

-  Had  previous  college 

-0.1227 

0.4910 

-0.9651 b  0.4746 

-0.2246 

0.4699 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

0.2201 

0.5103 

0.3240  0.4815 

0.9455c 

0.5131 

-  Gender  (female) 

0.6721 

0.4890 

0.7964c  0.4739 

0.6428 

0.4858 

-  Age  in  years 

-0.0200 

0.0588 

0.0467  0.0569 

0.0410 

0.0635 

-  Race  (not  Caucasian) 

0.4788 

0.4616 

0.81 79c  0.4282 

0.5645 

0.4409 

-  Married 

0.2777 

0.6136 

-0.2999  0.5590 

0.4380 

0.6008 

-  Has  children 

0.3465 

0.3316 

0.3706  0.3172 

0.2927 

0.3152 

-  Resides  with  family 

-0.4766 

0.6543 

-0.2096  0.6144 

-0.3464 

0.6164 

-  Lives  on  base  (in  San  Diego) 

-0.2716 

0.5413 

0.2496  0.5324 

-1.11 50b 

0.5473 

-  Lives  on  base  (in  Portsmouth) 

-0.0267 

0.8228 

-0.8418  0.6954 

0.3299 

0.7804 

Mean  predicted  rate  of  satisfaction11: 

-  TNCC 

0.92 

0.91 

0.89 

-  NSHS  San  Diego 

0.77 

0.67 

0.78 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 


d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ...  (continued) 


Monitoring  of  inst 

Access  to  inst 

Access  to  computer 

Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

3.2582 

3.5024 

-0.3558 

4.3884 

3.8350 

3.2044 

Treatment  group  (TNCC=1) 

1.2306a 

0.4826 

0.6684 

0.6179 

0.1962 

0.4057 

FY  1  999  (startup) 

-0.0399 

0.3941 

0.2126 

0.5247 

-0.951a 

0.3459 

Very  to  somewhat  satisfied 

with  orders  upon  graduation 

0.6214 

0.3883 

1.2230a 

0.4641 

0.2702 

0.3576 

Student  quality: 

-  A-school  GPA 

-0.0444 

0.0399 

-0.0094 

0.0507 

-0.0416 

0.0367 

-  A-school  GPA  is  above 

average  for  one's  classmates 

0.4966 

0.3755 

0.6150 

0.4705 

-0.0177 

0.3368 

-  Missing  some  prerequisite 

1.2457 

1.1212 

-0.1457 

1.1603 

-0.4177 

0.7849 

-  Had  previous  college 

-0.81 98b 

0.4032 

-0.6617 

0.4949 

-0.4296 

0.3589 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

-0.1042 

0.4134 

0.4908 

0.5203 

-0.1669 

0.3825 

-  Gender  (female) 

0.4712 

0.4062 

0.0636 

0.4872 

0.5832c 

0.3591 

-  Age  in  years 

0.0171 

0.0464 

0.0493 

0.0616 

-0.0157 

0.0414 

-  Race  (not  Caucasian) 

0.9471 a 

0.3731 

0.7932c 

0.4476 

0.5463c 

0.3384 

-  Married 

-0.1074 

0.4761 

0.2053 

0.5957 

0.5911 

0.4325 

-  Has  children 

-0.1311 

0.2535 

-0.1484 

0.3376 

-0.0289 

0.2268 

-  Resides  with  family 

0.2425 

0.5253 

0.8904 

0.6881 

0.1328 

0.4725 

-  Lives  on  base  (in  San  Diego) 

-0.0648 

0.4846 

-0.2814 

0.6106 

0.9568b 

0.4963 

-  Lives  on  base  (in  Portsmouth) 

-0.8268 

0.5352 

-0.7627 

0.6413 

-0.0667 

0.4446 

Mean  predicted  rate  of  satisfaction^ 

-  TNCC 

0.83 

0.89 

0.61 

-  NSHS  San  Diego 

0.61 

0.81 

0.57 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 


d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 


72 


Appendix 


Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ...  (continued) 


Quality  of  equip 

Quality  of  texts 

Clinical  rotations 

Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

7.0770 

4.7091 

8.9585 

6.1077 

8.5284a 

3.4835 

Treatment  group  (TNCC=1) 

-0.1797 

0.5945 

1.2261 

0.8147 

0.1108 

0.4270 

FY  1  999  (startup) 

1 .1 994b 

0.5479 

1.0090 

0.7631 

-0.2170 

0.3688 

Very  to  somewhat  satisfied 
with  orders  upon  graduation 

1 .1 1 2  6b 

0.4731 

0.9991 c 

0.5972 

0.87793 

0.3648 

Student  quality: 

-  A-school  GPA 

-  A-school  GPA  is  above 

-0.0984c 

0.0531 

-0.1 370b 

0.0681 

-0.12303 

0.0401 

average  for  one's  classmates 

0.2068 

0.4571 

0.7303 

0.5770 

0.731 9b 

0.3527 

-  Missing  some  prerequisite 

-1 .5271 c 

0.8497 

-0.4792 

1.2509 

-1.4785b 

0.7644 

-  Had  previous  college 

-0.7508 

0.4854 

-0.6117 

0.5966 

-0.3638 

0.3701 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

-0.1003 

0.5132 

-0.1677 

0.6270 

0.0468 

0.3973 

-  Gender  (female) 

0.9367c 

0.4921 

0.2413 

0.5759 

0.1561 

0.3686 

-  Age  in  years 

0.0910 

0.0626 

0.1445 

0.0937 

0.0764c 

0.0462 

-  Race  (not  Caucasian) 

0.4735 

0.4395 

1 .431  7a 

0.5577 

0.6636b 

0.3377 

-  Married 

1.0443c 

0.6224 

0.4669 

0.7686 

0.0377 

0.4464 

-  Has  children 

-0.4088 

0.3084 

-0.2598 

0.4010 

0.1108 

0.2426 

-  Resides  with  family 

0.0484 

0.6766 

0.1799 

0.8767 

-0.0427 

0.5046 

-  Lives  on  base  (in  San  Diego) 

-0.3112 

0.6015 

-0.8139 

0.6549 

0.1226 

0.4968 

-  Lives  on  base  (in  Portsmouth) 

-0.2858 

0.6225 

-0.2695 

1.0345 

-0.3254 

0.4644 

Mean  predicted  rate  of  satisfaction^ 

-  TNCC 

0.83 

0.94 

0.67 

-  NSHS  San  Diego 

0.85 

0.85 

0.65 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 


d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ...  (continued) 


Prep  for  clinical  Prep  to  be  MLT  Prep  for  ASCP  exam 


Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

1.2397 

3.5375 

-1.4725 

4.3102 

2.9769 

3.4189 

Treatment  group  (TNCC=1) 

0.9307b 

0.4531 

0.8986 

0.5774 

0.9481 b 

0.4510 

FY  1 999  (startup) 

0.3073 

0.4063 

0.1414 

0.4950 

0.2633 

0.3734 

Very  to  somewhat  satisfied 
with  orders  upon  graduation 

0.8362b 

0.3878 

0.961 5b 

0.4454 

0.3338 

0.3661 

Student  quality: 

-  A-school  GPA 

-  A-school  GPA  is  above 

-0.0333 

0.0403 

0.0017 

0.0494 

-0.0423 

0.0385 

average  for  one's  classmates 

0.5440 

0.3797 

-0.0383 

0.4414 

0.0427 

0.3569 

-  Missing  some  prerequisite 

0.2795 

0.8800 

-0.5804 

0.9586 

-0.5751 

0.7905 

-  Had  previous  college 

-0.7771 b 

0.4032 

-0.3834 

0.4552 

0.4328 

0.3680 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

-0.2781 

0.4236 

-0.3836 

0.4836 

-0.4674 

0.3953 

-  Gender  (female) 

0.1911 

0.4024 

0.5281 

0.4648 

0.5835 

0.3847 

-  Age  in  years 

0.0426 

0.0484 

0.0326 

0.0622 

-0.0027 

0.0456 

-  Race  (not  Caucasian) 

0.9043a 

0.3692 

1 .2631 a 

0.4374 

0.81 06b 

0.3584 

-  Married 

0.0145 

0.4824 

0.5052 

0.6117 

0.7191 

0.4768 

-  Has  children 

0.0637 

0.2624 

0.4067 

0.3722 

-0.0660 

0.2434 

-  Resides  with  family 

0.2132 

0.5351 

0.7431 

0.7078 

-0.2550 

0.5238 

-  Lives  on  base  (in  San  Diego) 

0.6189 

0.4974 

-0.4620 

0.5484 

-0.9205c 

0.4825 

-  Lives  on  base  (in  Portsmouth) 

0.2298 

0.5504 

0.1417 

0.6873 

-1 .001 7b 

0.5003 

Mean  predicted  rate  of  satisfaction0: 

-  TNCC 

0.82 

0.87 

0.72 

-  NSHS  San  Diego 

0.66 

0.76 

0.53 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 


d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 


74 


Appendix 


Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ...  (continued) 


Prep  for  A. A.  degree 

Fellow  students 

Discipline 

Overall 

Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

2.6424 

3.2898 

-2.8204 

3.2054 

1.5613 

3.4412 

-0.8297 

3.9763 

Treatment  group  (TNCC=1) 

0.4147 

0.4285 

0.6542 

0.4175 

-0.2610 

0.4511 

2.07063 

0.5881 

FY  1999  (startup) 

0.1272 

0.3623 

-0.5652 

0.3568 

0.3355 

0.3904 

-0.3381 

0.4583 

Very  to  somewhat  satisfied 

with  orders  upon  graduation 

0.5164 

0.3589 

0.6541 c 

0.3528 

0.4750 

0.3922 

1 .1 071 3 

0.4185 

Student  quality: 

-  A-school  GPA 

-  A-school  GPA  is  above 

-0.0340 

0.0374 

0.0159 

0.0366 

0.0024 

0.0397 

-0.0156 

0.0448 

average  for  one's  classmates 

0.5414 

0.3434 

-0.4368 

0.3360 

-0.1462 

0.3717 

0.1607 

0.4176 

-  Missing  some  prerequisite 

1.3607 

1.1150 

0.7025 

0.9108 

1.1909 

1.1231 

-0.4377 

0.8687 

-  Had  previous  college 

-0.5330 

0.3639 

-0.3018 

0.3611 

-0.6312 

0.4104 

-0.5782 

0.4357 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

0.1219 

0.3806 

0.8296° 

0.3867 

0.5431 

0.4200 

-0.3177 

0.4696 

-  Gender  (female) 

0.5575 

0.3656 

-0.2021 

0.3531 

0.6622° 

0.3966 

0.1195 

0.4499 

-  Age  in  years 

-0.0039 

0.0416 

0.0426 

0.0429 

-0.0850b 

0.0440 

0.0681 

0.0602 

-  Race  (not  Caucasian) 

0.2622 

0.3431 

0.7007b 

0.3427 

1 .21 33a 

0.3659 

1 .1 051 a 

0.4202 

-  Married 

0.5473 

0.4334 

-0.0540 

0.4269 

-0.3093 

0.4690 

0.7493 

0.5717 

-  Has  children 

-0.1697 

0.2347 

-0.1205 

0.2317 

0.4773° 

0.2644 

-0.1926 

0.2976 

-  Resides  with  family 

0.5123 

0.4810 

0.7222 

0.4761 

0.2884 

0.5208 

0.0339 

0.6060 

-  Lives  on  base  (in  San  Diego) 

-0.4131 

0.4641 

-0.1139 

0.4733 

-0.1475 

0.5416 

-0.0909 

0.5032 

-  Lives  on  base  (in  Portsmouth) 

-0.9003b 

0.4511 

-0.6362 

0.4572 

0.1202 

0.4981 

-0.5982 

0.6954 

Mean  predicted  rate  of  satisfaction11: 

-  TNCC  0.68  0.68  0.70  0.91 

-  NSHS  San  Diego  0.59  0.54  0.74  0.62 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 

d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Appendix  G:  Population  statistics  and  logistic 
regression  results  for  student  satisfaction  with 
quality  of  life  (end-of-course  survey) 


For  satisfaction  with  quality  of  life  (QOL),  we  compare  rates  of  satisfaction 
with  several  aspects  of  QOL  across  TNCC  MLT  graduates  and  selected  NSHS 
Portsmouth  (PTS)  C-school  graduates  (X-ray  and  Pharmacy  Technicians).  Sat¬ 
isfaction  was  measured  using  an  end-of-course  (EOC)  survey  instrument. 1 
Table  1  lists  those  questions  that  dealt  specifically  with  the  student’s  satisfac¬ 
tion  with  several  aspects  of  QOL  while  attending  C-school. 


Table  1.  End-of-course  survey  questions:  satisfaction  with  QOL 


Short  name 

Orientation: 
Attire  policy: 
Commute: 
PT  sessions: 
Study  sessions: 

Your  locker: 
Student  lounge: 
Welcome  aboard  packet: 
Access  to  meals: 
Access  to  health  care: 
Access  to  OTC  meds: 
Personal  time: 
Support  for  advancement: 
Handling  financial  issues: 
Overall  staff's  help: 


Survey  question,  "How  satisfied  are  you  with..." 

The  check-in  and  orientation  at  NSHS 
Uniform  and  civilian  attire  policies 

The  commute  between  your  living  quarters  and  the  school  house 
The  quantity  and  quality  of  group  PT  sessions 

The  quantity  and  quality  of  group  and  mandatory  night  study  sessions 
The  quality  of  and  accessibility  to  your  locker 
The  student  lounge,  including  the  refrigerator 

Your  Welcome  Aboard  packet,  including  info  about  the  school  and  the  community 

Access  to  adequate  meals 

Access  to  health  care,  including  sick  call 

Access  to  over-the-counter  medications 

The  amount  of  time  off  you  received  to  attend  to  personal  matters 
The  support  available  to  help  pursue  opportunities  for  military  advancement 
The  handling  of  your  pay,  allowances,  travel  claims,  and  other  financial  issues 
The  school  house  staff's  help  overall  in  meeting  your  needs 


1 .  NSHS  PTS  graduates  were  surveyed  at  the  end  of  their  course  (EOC).  TNCC  MLT 
students  were  surveyed  at  the  end  of  their  didactic  phase  (first  six  months  taught 
on  the  TNCC  campus)  and  again  at  the  EOC.  For  QOL,  we  compared  NSHS  PTS 
graduates’  EOC  responses  to  TNCC  graduates  responses  to  the  end-of-didactic 
phase  survey.  This  will  capture  TNCC  graduates  satisfaction  with  QOL  while 
attending  courses  off  of  the  NMC  PTS  base.  We  used  TNCC  responses  from  grad¬ 
uates  only  (therefore,  10  end-of-didactic  phase  survey  respondents — those  that  did 
not  graduate — were  deleted  from  the  population  sample). 
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Valid  responses  to  the  survey  questions  were  measured  on  a  5-point  Likert  scale 
(l=very  satisfied,  2=somewhat  satisfied,  3=neither  satisfied  nor  dissatisfied, 
4=somewhat  dissatisfied,  5=very  dissatisfied).  For  our  analysis,  satisfaction 
responses  were  collapsed  into  a  dichotomous  variable  where  satisfaction  was 
determined  to  be  a  response  of  somewhat  to  very  satisfied  (1  or  2)  versus  not 
satisfied  (a  response  of  3,  4,  or  5). 

Tables  2  and  3  provide  population  means  and  logistic  regression  results  for  each 
aspect  of  program  satisfaction. 


Table  2.  Population  means:  TNCC  and  NSHS  PTS  graduates  with  completed  EOC  survey 

NSHS  PTS  TNCC  Pooled 
Variable  (n=261)  (n=136)  (n=397) 


Demographics: 


-  Paygrade  of  E-l  -  E-3 

0.45 

0.38 

0.42 

-  Gender  (female)3 

0.29 

0.39 

0.33 

-  Married 

0.49 

0.42 

0.46 

-  Has  children 

0.37 

0.35 

0.37 

-  Resides  with  family 

0.34 

0.32 

0.34 

-  Lives  off  base3 

0.41 

0.56 

0.46 

a.  Difference  in  population  means  is  statistically  significant. 
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Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ... 

Orientation  Attire  policy  Commute 


Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

0.7212° 

0.2874 

1 .4884° 

0.3111 

1 .0237° 

0.2814 

Treatment  group  (TNCC=1) 

-0.4291 

0.3100 

-0.7292a 

0.3224 

0.3423 

0.3154 

FY  1  999  (startup) 

0.3083 

0.2378 

-0.2657 

0.2402 

-0.3330 

0.2206 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

-0.1398 

0.2607 

-0.0595 

0.2736 

-0.761 3a 

0.2487 

-  Gender  (female) 

0.0576 

0.2612 

0.1369 

0.2723 

-0.1418 

0.2476 

-  Married 

0.6991b 

0.3079 

0.0763 

0.3208 

-0.0896 

0.2852 

-  Has  children 

-0.3300 

0.2935 

0.0128 

0.2960 

-0.0419 

0.2718 

-  Resides  with  family 

-0.0014 

0.3338 

-0.691 7b 

0.3388 

-0.0656 

0.3048 

-  Lives  off  base  and  attends  TNCC 

0.0059 

0.3827 

0.1317 

0.3877 

-0.3619 

0.3904 

Mean  predicted  rate  of  satisfaction0: 

-  TNCC  0.64  0.62  0.67 

-NSHSPTS  0.73  0.77  0.61 


a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 

d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ...  (continued) 

PT  sessions  Study  sessions  Your  locker 

Variable  Coefficient  S.E.  Coefficient  S.E.  Coefficient  S.E. 


Constant 

-0.8400a  1 

Treatment  group  (TNCC=1) 

0.97753  1 

FY  1  999  (startup) 

0.2408  l 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

-0.0563 

-  Gender  (female) 

-0.4019 

-  Married 

0.51 98c 

-  Has  children 

0.4336 

-  Resides  with  family 

-0.4419 

-  Lives  off  base  and  attends  TNCC 

0.0695 

Mean  predicted  rate  of  satisfaction0: 

-TNCC 

0.58 

-  NSHS  PTS 

0.35 

1 ,0489a 

0.2981 

0.96843 

0.3358 

-1.4333a 

0.3105 

1.0082b 

0.4644 

-0.2588 

0.2371 

-0.2435 

0.2734 

0.0602 

0.2664 

0.3140 

0.3142 

0.2173 

0.2692 

0.1086 

0.3109 

0.4100 

0.3109 

0.5779 

0.3652 

-0.3616 

0.2949 

-0.2838 

0.3485 

0.0163 

0.3296 

0.4848 

0.3913 

-0.3666 

0.3891 

-0.7992 

0.5469 

0.40  0.89 

0.75  0.79 


0.2796 

0.3093 

0.2226 

0.2509 

0.2508 

0.2868 

0.2748 

0.3055 

0.3785 


a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 

d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ...  (continued) 


Student  lounge  Welcome  aboard  packet  Access  to  meals 


Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

0.4659c 

0.2828 

-0.89443 

0.2793 

0.0354 

0.2786 

Treatment  group  (TNCC=1) 

0.7857b 

0.3299 

0.9701 a 

0.3054 

1.11543 

0.3260 

FY  1999  (startup) 

-0.92763 

0.2298 

0.0693 

0.2216 

-0.6871 a 

0.2282 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

0.3073 

0.2573 

0.3648 

0.2486 

-0.1998 

0.2530 

-  Gender  (female) 

-0.5796b 

0.2543 

-0.0007 

0.2464 

-0.2338 

0.2548 

-  Married 

0.1379 

0.2919 

0.3705 

0.2847 

0.2654 

0.2927 

-  Has  children 

0.3113 

0.2832 

0.2905 

0.2724 

0.3402 

0.2771 

-  Resides  with  family 

0.3094 

0.3132 

-0.0823 

0.3038 

0.2377 

0.3087 

-  Lives  off  base  and  attends  TNCC 

-0.0932 

0.4095 

0.1830 

0.3797 

0.2037 

0.4180 

Mean  predicted  rate  of  satisfaction15: 

-  TNCC  0.73 

-  NSHS  PTS  0.58 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 

d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 


0.63  0.74 

0.39  0.48 
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Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ...  (continued) 

Access  to  health  care  Access  to  OTC  meds  Personal  time 


Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

-0.2086 

0.2690 

0.2975 

0.2672 

1.3476a 

0.2938 

Treatment  group  (TNCC=1) 

0.1572 

0.3040 

-0.0512 

0.2970 

-0.73443 

0.3059 

FY  1 999  (startup) 

-0.41 29c 

0.2215 

-0.4070c 

0.2157 

-0.1201 

0.2299 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

0.0484 

0.2443 

0.1045 

0.2410 

-0.77963 

0.2562 

-  Gender  (female) 

-0.74833 

0.2493 

-0.2830 

0.2389 

-0.2817 

0.2560 

-  Married 

0.1141 

0.2799 

0.0937 

0.2756 

-0.0639 

0.2964 

-  Has  children 

0.1299 

0.2650 

0.1995 

0.2648 

-0.0756 

0.2853 

-  Resides  with  family 

0.3909 

0.2962 

0.2366 

0.2958 

0.2254 

0.3217 

-  Lives  off  base  and  attends  TNCC 

0.2627 

0.3750 

-0.0846 

0.3684 

0.2873 

0.3795 

Mean  predicted  rate  of  satisfaction0: 

-TNCC 

0.46 

0.56 

0.55 

-  NSHS PTS 

0.41 

0.57 

0.70 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 


d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Table  3.  Logistic  regression  results:  somewhat  to  very  satisfied  with  ...  (continued) 

Support  for  advancement  Handling  financial  issues  Overall  staffs'  help 


Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

0.80793 

0.2774 

0.0580 

0.2666 

0.91 08a 

0.3059 

Treatment  group  (TNCC=1) 

-0.7775a 

0.3054 

-0.0270 

0.3013 

0.5484 

0.3704 

FY  1 999  (startup) 

-0.1988 

0.2194 

-0.3068 

0.2174 

0.3521 

0.2539 

Demographics: 

-  Paygrade  of  E-1  -  E-3 

-0.5069b 

0.2483 

-0.1771 

0.2416 

-0.2045 

0.2782 

-  Gender  (female) 

-0.4895b 

0.2436 

-0.3147 

0.2423 

-0.0783 

0.2754 

-  Married 

0.0856 

0.2824 

-0.3622 

0.2803 

0.1730 

0.3192 

-  Has  children 

-0.4210 

0.2723 

-0.0588 

0.2644 

-0.2416 

0.3062 

-  Resides  with  family 

0.2889 

0.3016 

0.1720 

0.2977 

0.2837 

0.3425 

-  Lives  off  base  and  attends  TNCC 

-0.2163 

0.3822 

0.3813 

0.3697 

-0.3426 

0.4561 

Mean  predicted  rate  of  satisfaction0: 

-TNCC 

0.39 

0.42 

0.81 

-  NSHS PTS 

0.58 

0.41 

0.73 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 


d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Appendix  H:  Population  statistics  and  logistic 
regression  results  for  post-graduation  survey  of 
MLT  graduates 

The  post-graduation  survey  of  MLT  graduates  was  designed  to  measure  a 
graduate’s  satisfaction  with  how  well  the  MLT  program  prepared  him/her 
to  be  a  Navy  MLT  at  his/her  first  duty  station  (would  one’s  satisfaction  with 
the  program  change  over  time?).  Using  the  post-graduation  survey  of  MLT 
graduates,  we  compare  rates  of  satisfaction  with  the  MLT  program  across 
TNCC  MLT  graduates  and  NSHS  San  Diego  MLT  graduates.  Survey  ques¬ 
tions  are  shown  in  table  1 . 


Table  1.  Post-graduation  survey  of  MLT  graduates 
Short  name  Survey  question 

Knowledge:  In  your  opinion,  how  well  did  the  Navy's  MLT  Program 

Prepare  you  in  the  knowledge  required  of  an  entry-level  MLT? 
(l=well  above  satisfactory  /5=well  below  satisfactory) 

Technical  skills:  In  your  opinion,  how  well  did  the  Navy's  MLT  Program 

Prepare  you  in  the  technical  skills  required  to  perform  as  an  an 
entry-level  MLT? 

(l=well  above  satisfactory  /5=well  below  satisfactory) 


Military  behavior:  After  spending  a  year  in  the  MLT  program,  did  you  find  it 

difficult  to  understand  and  adjust  to  the  standards  of  military 
behavior,  bearing,  and  discipline  that  were  expected  of  you? 
(1=very  diff icu lt/5=very  easy) 


Valid  responses  to  the  survey  questions  were  measured  on  a  5 -point  Likert 
scale  (l=very  satisfied/difficult,  2=somewhat  satisfied/difficult,  3=neither 
satisfied/difficult  nor  dissatisfied/easy,  4=somewhat  dissatisfied/easy, 
5=very  dissatisfied/easy).  For  our  analysis,  satisfaction  responses  were  col¬ 
lapsed  into  a  dichotomous  variable  where  satisfaction  was  determined  to  be 
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a  response  of  somewhat  to  very  satisfied  (or  easy)  versus  not  satisfied  (or 
not  easy). 

The  graduate  survey  was  conducted  by  telephone  4  months  and  12  months 
after  graduation.  To  date,  we  have  completed  the  4-month  survey  for  all 
MLT  graduates  in  the  pilot  study.  The  12-month  surveys  are  still  being  con¬ 
ducted.  Therefore,  tables  2  and  3  provide  population  statistics  and  logistic 
regression  results  for  the  4-month  survey  only. 


Table  2.  Population  means:  TNCC  and  NSHS  SD  graduates  with  completed 
4-month  post-graduation  survey 


Variable 

NSHS  SD 
(n=109) 

TNCC 

(n=130) 

Pooled 

(n=239) 

Student  quality: 

-  AFQT  score 

64.20 

65.34 

15.04 

-  A-school  GPA 

88.18 

88.41 

4.40 

-  Had  previous  college 

0.58 

0.66 

0.48 

Demographics: 

-  Gender  (female)3 

0.23 

0.38 

0.49 

-  Race  (not  Caucasian) 

0.67 

0.65 

0.48 

-  Age  in  years 

23.69 

24.51 

4.39 

-  Married 

0.41 

0.45 

0.50 

-  Has  children 

0.31 

0.38 

0.49 

Quality  of  peers: 

-  Average  AFQT  of  classmates3 

62.82 

65.15 

1.74 

a.  Difference  in  population  means  is  statistically  significant. 
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Table  3.  Logistic  regression  results:  4-month  post-graduation  survey  of  graduates 


Knowledge  Technical  skills  Military  behavior 


Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

8.0956 

6.6195 

5.5270 

7.7514 

5.1490 

6.9571 

Treatment  group  (TNCC=1) 

0.2940 

0.3702 

0.6203 

0.4283 

0.2163 

0.3901 

FY  1 999  (startup) 

0.4352 

0.3470 

0.7453c 

0.4120 

0.7051b 

0.3662 

Student  quality: 

-  AFQT  score 

0.0002 

0.0103 

-0.0038 

0.0116 

0.0044 

0.0107 

-  A-school  GPA 

-0.0245 

0.0355 

-0.0098 

0.0395 

-0.0254 

0.0367 

-  Had  previous  college 

0.3258 

0.3249 

-0.1562 

0.3696 

0.3196 

0.3393 

Demographics: 

-  Gender  (female)3 

-0.2042 

0.3375 

-0.5415 

0.3651 

-0.0461 

0.3479 

-  Race  (not  Caucasian) 

-0.4732 

0.3307 

0.0107 

0.3587 

-1.23893 

0.3728 

-  Age  in  years 

0.0083 

0.0343 

-0.0364 

0.0364 

0.0181 

0.0363 

-  Married 

-0.3294 

0.3629 

-0.2077 

0.3985 

0.0321 

0.3783 

-  Has  children 

-0.3829 

0.3744 

-0.2444 

0.4120 

-0.1398 

0.3931 

Quality  of  peers: 

-  Ave.  AFQT  of  classmates3 

-0.0828 

0.0973 

-0.0388 

0.1142 

-0.0389 

0.1021 

Mean  predicted  rate  of  satisfaction/easec 

i. 

-  TNCC 

0.69 

0.80 

0.70 

-  NSHS  San  Diego 

0.93 

0.70 

0.66 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 


d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Appendix  I:  Population  statistics  and  logistic 
regression  results  for  post-graduation  survey  of 
MLT graduates’  supervisors 

The  post-graduation  survey  of  MLT  graduates’  immediate  supervisors  was 
the  instrument  used  to  measure  the  “end-users’”  satisfaction  with  MLT 
graduates  in  terms  of  knowledge,  technical  skills,  military  behavior,  job 
attitude,  and  overall  performance.  Survey  questions  are  shown  in  table  1. 


Table  1.  Post-graduation  survey  of  supervisors 
Short  name  Survey  question 

Knowledge:  In  your  opinion,  how  well  did  the  Navy's  MLT  Program  prepare  this  sailor  in 
the  knowledge  required  of  an  entry-level  MLT? 

(l=well  above  satisfactory  /5=well  below  satisfactory) 

Technical  skills:  In  your  opinion,  how  well  did  the  Navy's  MLT  Program  prepare  this  sailor  in 
the  technical  skills  required  to  perform  as  an  an  entry-level  MLT? 

(l=well  above  satisfactory  /5=well  below  satisfactory) 

Military  behavior:  In  your  opinion,  does  this  sailor  adhere  to  appropriate  standards  of  military 
behavior,  bearing,  and  discipline? 

(1=well  above  satisfactory  /5=well  below  satisfactory) 


Attitude:  In  your  opinion,  does  this  sailor  exhibit  the  proper  attitude  toward  the  job? 
(l=well  above  satisfactory  /5=well  below  satisfactory) 


Performance  rank:  Overall,  in  comparison  to  other  entry-level  MLTs,  how  well  is  this  sailor 
performing? 

(1=well  above  satisfactory  /5=well  below  satisfactory) 


Valid  responses  to  the  survey  questions  were  measured  on  a  5 -point  Likert 
scale  (l=well  above  satisfactory,  2=above  satisfactory,  3=satisfactory, 
4=below  satisfactory,  5=well  below  satisfactory).  For  our  analysis,  super¬ 
visor’s  responses  were  collapsed  into  a  dichotomous  variable  where  a  value 


89 


Appendix 


of  1  indicates  above  satisfactory  performance  (supervisor  response  of  1  or 
2)  and  0  indicates  satisfactory  to  well  below  satisfactory  performance  (a 
supervisor  response  of  3,  4,  or  5). 

Supervisors  of  MLT  graduates  from  TNCC  and  NSHS  San  Diego  were  sur¬ 
veyed  4  months  and  12  months  after  graduation.  To  date,  we  have  com¬ 
pleted  the  4-month  survey  for  all  MLT  graduates  in  the  pilot  study.  The  12- 
month  surveys  are  still  being  conducted.  Therefore,  this  appendix  provides 
population  statistics  and  logistic  regression  results  for  the  4-month  survey 
only.  These  are  presented  in  tables  2  and  3. 


Table  2.  Population  means:  TNCC  and  NSHS  SD  graduates  with  completed 
4-month  post-graduation  survey  of  supervisors 


Variable 

NSHS  SD 
(n=112) 

TNCC 

(n=134) 

Pooled 

(n=246) 

Student  quality: 

-  AFQT  score 

64.25 

65.04 

64.68 

-  A-school  GPA 

88.16 

88.42 

88.30 

-  Had  previous  college 

0.58 

0.66 

0.62 

Demographics: 

-  Gender  (female)3 

0.24 

0.40 

0.33 

-  Race  (not  Caucasian) 

0.68 

0.66 

0.67 

-  Age  in  years 

23.71 

24.47 

24.13 

-  Married 

0.39 

0.46 

0.43 

-  Has  children 

0.32 

0.39 

0.36 

Quality  of  peers: 

-  Average  AFQT  of  classmates3 

62.78 

65.13 

64.06 

a.  Difference  in  population  means  is  statistically  significant. 
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Table  3.  Logistic  regression  results:  4-month  post-graduation  survey  of  supervisors 


Knowledge  Technical  skills  Military  behavior 


Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

-5.6483 

6.1798 

7.7650 

6.2469 

-3.8784 

7.6404 

Treatment  group  (TNCC=1) 

0.2191 

0.3472 

0.4317 

0.3514 

-0.1864 

0.4358 

FY  1 999  (startup) 

-0.2590 

0.3249 

0.3732 

0.3270 

-0.0213 

0.4000 

Student  quality: 

-  AFQT  score 

0.0242b 

0.0102 

0.0145 

0.0099 

0.0045 

0.0122 

-  A-school  GPA 

0.0075 

0.0338 

0.0079 

0.0336 

0.0076 

0.0417 

-  Had  previous  college 

-0.3866 

0.3154 

-0.3948 

0.3148 

-0.1992 

0.3850 

Demographics: 

-  Gender  (female)3 

0.2588 

0.3273 

0.1808 

0.3201 

0.0459 

0.3948 

-  Race  (not  Caucasian) 

0.86373 

0.3174 

0.4760 

0.3130 

-0.4470 

0.4090 

-  Age  in  years 

0.0347 

0.0346 

0.0296 

0.0342 

0.0924c 

0.0482 

-  Married 

0.2102 

0.3457 

-0.1091 

0.3438 

0.0362 

0.4249 

-  Has  children 

0.0318 

0.3578 

0.7354b 

0.3621 

0.1764 

0.4475 

Quality  of  peers: 

-  Ave.  AFQT  of  classmates3 

0.0397 

0.0905 

-0.1 624c 

0.0929 

0.0417 

0.1110 

Mean  predicted  rate  of  above  satisfactory  responses'1: 

-  TNCC 

0.64 

0.64 

0.80 

-  NSHS  San  Diego 

0.59 

0.55 

0.82 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 


d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Table  3.  Logistic  regression  results:  4-month  post-graduation  survey  of  supervisors  (continued) 

Attitude  Performance  rank 


Variable 

Coefficient 

S.E. 

Coefficient 

S.E. 

Constant 

-6.5736 

8.0994 

-11.3862c 

6.7839 

Treatment  group  (TNCC=1) 

-0.5958 

0.4743 

-0.1496 

0.3827 

FY  1 999  (startup) 

0.0828 

0.4211 

0.5282 

0.3476 

Student  quality: 

-  AFQT  score 

0.0128 

0.0131 

0.01 99c 

0.0108 

-  A-school  GPA 

0.0592 

0.0438 

0.0697b 

0.0358 

-  Had  previous  college 

-0.1939 

0.4033 

0.0174 

0.3284 

Demographics: 

-  Gender  (female)3 

0.2183 

0.4199 

0.4557 

0.3441 

-  Race  (not  Caucasian) 

-0.4663 

0.4372 

0.2446 

0.3338 

-  Age  in  years 

0.0982c 

0.0517 

0.0360 

0.0368 

-  Married 

-0.0305 

0.4437 

0.2492 

0.3630 

-  Has  children 

0.2692 

0.4676 

-0.3505 

0.3738 

Quality  of  peers: 

-  Ave.  AFQT  of  classmates3 

0.0072 

0.1169 

0.0547 

0.0989 

Mean  predicted  rate  of  above  satisfactory  responses'1: 

-  TNCC 

0.79 

0.67 

-  NSHS  San  Diego 

0.87 

0.70 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 


d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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Appendix  J:  Population  statistics  and  logistic 
regression  results  for  incidence  of  disciplinary 
action  comparison 

To  evaluate  whether  the  MLT  pilot  program  had  any  deleterious  effect  on 
military  bearing  of  its  students,  we  use  several  measures,  one  being  the  inci¬ 
dence  of  disciplinary  actions.  For  this  measure,  we  compare  the  rate  of 
disciplinary  actions  that  led  to  the  generation  of  a  Minor  Offense  Report 
(MOR)  or  higher  (e.g.,  Disciplinary  Review  Board,  XOI,  Mast)  among 
TNCC  and  NSFIS  San  Diego  MLT  students.  Tables  1  and  2  provide  popu¬ 
lation  means  and  logistic  regression  results. 


Table  1:  Population  means  for  TNCC  and  NSHS  SD  unique  enrolleesa 

NSHS  SD  TNCC  Pooled 
Variable  (n=159)  (n=150)  (n=309) 


Disciplinary  action  rate 

0.07 

0.07 

0.07 

Student  quality: 

-  AFQT  score 

63.94 

64.99 

64.45 

-  A-school  GPA 

87.09 

87.88 

87.47 

-  Missing  some  prerequisite1’ 

0.09 

0.03 

0.06 

-  Had  previous  college1’ 

0.53 

0.63 

0.58 

Demographics: 

-  Paygrade  of  E-1  to  E-3b 

0.60 

0.34 

0.47 

-  Gender  (female)13 

0.21 

0.36 

0.28 

-  Race  (not  Caucasian) 

0.69 

0.66 

0.67 

-  Married 

0.37 

0.43 

0.40 

-  Has  children 

0.30 

0.37 

0.33 

-  Age  in  years 

23.47 

24.17 

23.81 

a.  The  population  of  unique  enrollees  counts  set  backs  only  once  (thus  enrollments 


that  end  in  a  set  back  are  excluded  from  the  sample), 
b.  Difference  in  population  means  is  statistically  significant. 
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Table  2.  Logistic  regression  results:  incidence  of  disciplinary  action 


Variable 

Coefficient 

Standard  error 

Constant 

-2.3576 

4.7571 

Treatment  group  (TNCC=1 ) 

0.7714 

0.5243 

FY  1 999  (startup) 

-1 .51 73b 

0.6513 

Student  quality: 

-  AFQT  score 

-0.0050 

0.0185 

-  A-school  GPA 

0.0050 

0.0559 

-  Missing  some  prerequisite 

0.2895 

0.8534 

-  Had  previous  college 

-0.1421 

0.5479 

Demographics: 

-  Paygrade  of  E-1  to  E-3 

0.8721 

0.6310 

-  Gender  (female) 

-1.6252b 

0.8021 

-  Race  (not  Caucasian) 

0.6895 

0.6029 

-  Married 

-0.6670 

0.6647 

-  Has  children 

0.4401 

0.6901 

-  Age  in  years 

-0.0391 

0.0693 

Mean  predicted  disciplinary  action  rated: 

-  TNCC 

0.10 

-  NSHS  San  Diego 

0.05 

a.  Coefficient  significant  at  the  .01  level. 

b.  Coefficient  significant  at  the  .05  level. 

c.  Coefficient  significant  at  the  .10  level. 

d.  Differences  in  predicted  rates  of  satisfaction  associated  with  the  two  MLT  programs  are 
determined  to  be  statistically  significant  if  the  "treatment"  coefficient  is  significant. 
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FORCMJ.  L.  DiRosa,  OOHC 

RADM  A.  Diaz,  MED-05 

CAFT  L.  Daehn,  MED-05B 

LT  B.  Fletcher,  MED-51 

Ms.  M.  Lewis,  MED-51  A 

CAFT  G.  Mukri,  MED-53 

Ms.  B.  Wright,  MED-533 

Mr.  W.  Lorenzen,  MED-01B 

RADM  N.  Lescavage,  MED-03 

CAFT  P.  Bull,  MED-82 

CAFT  B.  Brannman,  N931 

CAPT  K.  Senn,  N132D13 

CAPT  C.  Mount,  CO,  NSEIS  Portsmouth 

LCDR  T.  Delucia,  MLT  Dept.  Elead,  NSEIS  Portsmouth 

Ms.  L.  Dezern,  MIT  Program  Director,  TNCC 

CAPT  M.  Kowba,  CO,  NSHS  San  Diego 
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LCDR  V.  Sullivan-Frederick,  MIT  Dept.  Head,  NSHS  San  Diego 
CAPT  D.  Wade,  CO,  NSHS  Bethesda 
CAPT  W.  Nunns,  XO,  NSHS  Bethesda 

LCDR  H.  Vandenburg,  Head,  Laboratory  Dept.,  NMCL  Quantico 
LCDR  G.  Mosman,  Head,  Laboratory  Dept.,  NH  Cherry  Point 
LCDR  C.  Congdon,  Head,  Laboratory  Dept.,  NH  Camp  Lejeune 
LCDR  W.  G.  Knight ,  DBSS  Project  Office,  Health  Affairs 
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